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The  Hughes  Aircraft  Corporation  and  the  RCA  Corporation  were  assigned 
complementary  tasks  by  the  Naval  Electronics  Laboratory  Center  under  two 
separate  contracts  for  studies  to  analyze  the  acquisition  of  automatic  test 
equipment  (ATE)  systems.  It  is  NELC's  intent  to  integrate  into  a single 
document  the  separate  final  study  reports  required  to  be  submitted  by  each 
company.  Toward  that  end,  NELC  issued  a combined  outline,  and  tasks  were 
assigned  to  each  company  within  that  framework.  This  report  represents  only 
the  results  of  the  RCA  effort.  However,  as  an  aid  to  visualizing  the  RCA  tasks 
within  the  total  context,  the  table  of  contents  and  the  body  of  the  report  are 
annotated  to  indicate  the  sections  which  were  assigned  to  Hughes. 
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SECTION  1 


INTRODUCTION1 


1.1  GENERAL 

The  potential  logistic  advantages  of  automatic  test  equipment  (ATE)  are  well 
understood.  This  study  has  identified  program  management  approaches  and 
procedures  toward  achieving  those  advantages,  with  particular  stress  on  the 
need  for  timeliness  in  making  ATE  decisions  and  on  the  organization  of  technical 
and  management  forces  to  implement  the  decision  process.  The  selection  of 
automatic  test  equipment  is  properly  a part  of  logistic  planning;  therefore,  the 
results  of  the  study  are  keyed  to  the  prime  equipment  acquisition  phases 
defined  in  NAVMAT  INST  4000. 20A,  titled  "Integrated  Logistic  Support  Planning 
Policy".2 

Some  of  the  information  presented  herein  is  already  well  known  to  ATE  experts; 
some  of  it  will  he  familiar  to  logisticians,  and  other  portions  will  he  obvious 
to  program  managers.  However,  this  is  the  first  time  that  all  of  this  information 
has  been  brought  together  for  the  guidance  of  SYSCOMS  in  the  future  selection  of 
ATE  - a process  which  requires  the  carefully  timed  cooperation  of  technical, 
logistic,  and  management  personnel. 


1Hughes  Aircraft  Corporation  inputs  were  integrated  by  RCA  into  this  combined 
section,  per  agreement  at  NELC  review  meeting,  73  Nov.  14. 

2 

It  is  recommended  that  the  procedural  portions  of  this  report  should  be  con- 
densed to  approximately  5 pages  and  prepared  in  a format  which  will  be 
acceptable  to  the  custodian  of  NAVMAT  INST  4000. 20A  as  an  appendix  to  that 
instruction. 
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Figure  1-1  diagrams  the  technical,  logistic,  and  management  interdisciplinary 
relationships  for  the  purpose  of  showing  how  ATE  selection  fits  in  with  the 
overall  prime  system  definition.  The  first  step  is  a technical  definition  of  the 
prime  equipment  to  be  supported,  based  on  operational  mission  requirements. 
This  information  is  needed  by  logisticians  to  enable  them  to  develop  support 
concepts , including  spares  and  repair  level  policies  upon  which  ATE  may 
impinge.  The  same  information  enables  the  test  equipment  engineer  to  develop 
test  requirements  upon  which  his  ATE  recommendations  will  eventually  be 


Figure  1-1.  Interdisciplinary  Relationships  of  ATE  Selection 


1-2 


based.  A double-headed  arrow  between  logistic  plans  and  ATE  concept/equip- 
ment description  denotes  the  iterative  nature  of  test  equipment  selection, 
whereby  initial  logistic  goals  may  be  modified  by  later  evaluation  of  tost  equip- 
ment capabilities. 1 The  prime  equipment  definition  and  the  logistic  plans, 
which  would  include  ATE  recommendations , would  then  be  reviewed  by  the 
SYSCOM  and  modified  if  necessary  for  compatibility  with  mission,  manning, 
cost,  and  schedule  requirements.  Finally,  a combined  prime  system  and 
support  concept  would  be  defined  by  the  SYSCOM,  This  same  general  procedure 
is  followed  at  every  phase  of  the  prime  equipment  acquisition  process,  except 
that  the  ILS  concept  and  the  ATE  can  be  defined  in  increasing  detail  as  the  prime 
equipment  design  matures  with  each  subsequent  phase.  Five  phases  of  prime 
equipment  development,  as  defined  in  Appendix  H of  NAVMAT  INST  4000. 20A, 
are  used  to  determine  the  various  events  and  decision  points  of  the  ATE  develop- 
ment cycle.  These  are: 

(1)  Conceptual  Phase 

(2)  Validation  Phase  (Engineering  and  Development) 

(3)  Full  Scale  Development  Phase  (Operational  System  Development) 

(4)  Production  Phase 

(5)  Deployment/Operational  Phase  (Logistic  Support,  Inventory  Control, 
Training,  etc.) 

During  the  Conceptual  Phase,  the  prime  equipment  is  defined  basically  by  needs 
and  objectives.  Very  general  support  requirements  are  specified,  such  as 


Although  this  study  is  devoted  entirely  to  automatic  test  equipment,  it  is 
recognized  that  manual  test  equipment  may  be  a”  attractive  alternative  for 
certain  limited  applications. 
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desired  (or  required)  operational  availability.  Comprehensive  system  studies 
are  performed. 

During  the  Validation  Phase,  the  major  program  characteristics  are  validated. 
This  phase  ends  with  the  contract  to  proceed  with  the  detail  design.  Support 
alternatives,  including  off-line  vs  on-line,  are  considered  and  general  require- 
ments for  support  equipment  are  resolved.  Logistic  support  models  which 
consider  system  life  cycle  cost  are  applicable  to  this  phase. 

During  the  Full  Scale  Development  Phase  the  prime  equipment  design  is 
completed,  and  the  support  equipment  is  defined  and  prototyped. 

During  the  Production  Phase  off-line  test  equipment  specifications  are  completed 
and  procurements  initiated. 

During  the  Deployment/Operational  Phase,  prime  equipment  specifications  are 
generally  frozen,  and  if  new  test  equipment  procurements  are  made,  they  must 
be  tailored  to  the  existing  prime  equipment. 

1 . 2 IMPACT  OF  ACQUISITION  PHASES 

For  homogeneous  equipment  or  systems , defined  as  those  in  which  all  components 
are  at  a similar  stage  of  development,  the  ATE/ILS  iterative  relationship  is 
essentially  as  shown  in  Figure  1-2,  which  is  an  expansion  of  the  ATE/ILS  portion 
of  Figure  1-1.  The  process  starts  with  the  initial  proposal  of  an  ILS  concept, 
which  will  include,  among  other  things,  availability,  manning  budgets,  and  the 
repair  policy  at  each  maintenance  echelon.  From  these  parameters , ATE 
alternatives  toward  meeting  ILS  goals  can  be  defined,  and  then  comparatively 
evaluated  to  establish  benefit/cost  ratios.  As  a result  of  this  evaluation  it  may 


Figure  1-2.  Interrelationship  of  ILS  and  ATE 

prove  necessary  to  feed  changes  back  into  the  initial  ILS  concept,  where  the 
ILS  concept  may  have  been  either  too  ambitious  or  not  demanding  enough.  For 
example  a module  tester  may  be  available  or  producible  at  a low  enough  cost 
to  make  it  economical  to  change  a logistic  policy  from  one  which  originally 
called  for  module  repair  at  depot  to  a policy  of  module  repair  on  shipboard, 
with  a saving  in  shipboard  spares  and  eliminations  of  depot  pipeline  delays. 

For  ships,  aircraft,  and  other  systems  which  are  heterogeneous  in  that  they 
utilize  items  in  various  stages  of  development,  the  process  shown  in  Figure  1-2 
becomes  operative  at  more  than  one  equipment  or  indenture  level  (e.  g.  component , 


module,  assembly,  etc.)  for  each  maintenance  echelon.  Figure  1-3  summarizes 
the  essentials  of  the  test  equipment  selection  process  for  such  a system,  consist- 


1-5 


Figure  1-3.  Impact  of  Mixed  Acquisition  Phases 

and  equipment  still  in  the  concept  phase.  For  prime  equipment  in  the  first  two 
categories , planned  and  existing  special  test  equipment  must  be  reviewed  for 
compatibility  with  the  proposed  ILS  concept.  If  not  compatible,  then  other  test 
equipment  alternatives  must  be  examined,  and  either  another  alternative  selected, 
or  the  ILS  concept  altered.  For  heterogeneous  systems,  it  may  be  necessary  to 
accept  compromises  which  result  in  a non-uniform  level  of  repair,  so  that  at  the 
organizational  echelon,  for  example,  some  equipments  may  be  repairable  by 
assembly  replacement,  some  by  module,  and  others  by  component  replacement. 
Test  methods  and  maintenance  procedures  might  be  compromised  instead,  in 
order  to  preserve  a uniform  repair  policy,  in  which  caf  e ATE  and  manual  test 
methods  plus  a nix  of  different  maintenance  skill  levels  might  be  employed  to 
accommodate  the  required  degree  of  fault  diagnosis. 
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If  test  equipment  originally  intended  for  prime  equipment  already  in  existence 
or  in  development  is  compatible  with  the  planned  ILS  concept,  then  that  same 
test  equipment  should  be  analyzed  to  determine  whether  it  can  also  service  other 
equipments  in  the  ship,  aircraft,  or  overall  prime  system,  and,  if  so,  whether 
modifications  are  required  for  that  purpose.  It  is  obvious  that  equipment  not 
yet  in  development  affords  the  SYSCOM  a full  range  of  ATE  options,  while  for 
equipment  further  along,  hard  choices  may  be  indicated  between  costly  modi- 
fications to  meet  ILS  goals,  and  compromising  those  goals.  Since  the  available 
ATE  options  rapidly  diminish  as  equipment  leaves  the  Validation  Phase,  it  is 
clear  that  the  maximum  benefits  of  ATE  car.  best  be  realized  through  planning 
which  starts  with  formulation  of  the  program  concept.  Timeliness  is  especially 
critical  when  planning  for  built-in  or  other  on-iine  test  equipment  configurations 
which  require  that  anywhere  from  a portion  to  all  of  the  test  equipment  will  be 
an  integral  part  of  the  prime  equipment.  Clearly,  deferring  the  decision  to  use 
built-in  or  on-line  test  until  prime  equipment  design  is  under  way  entails  the 
risk  of  a custly  design  change. 

The  selection  of  off-line  test  equipment  can  be  deferred  to  the  Production  or 
even  to  the  Development/Operational  Phase  with  the  potential  risk  limited  to 
obtaining  less  than  optimum  test  performance  where  the  designer  has  not  been 
required  to  optimize  test  point  accessibility  for  use  with  ATE. 

1.3  SKILLS  REQUIRED  FOR  ATE  SELECTION 

The  need  for  ATE  decisions  to  be  made  early  in  an  acquisition  program  can  only 
be  satisfied  by  a technical  team  which  contains  a mix  of  prime  equipment  and 
ATE  skills.  The  prime  system  is  not  normally  defined  in  sufficient  detail 
during  the  Validation  Phase  to  facilitate  ATE  selection,  except  where  exising 
equipment  is  being  used.  The  team  must  be  capable,  then,  of  synthesizing  the 
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proposed  system  in  the  detail  required  to  enable  test  requirements  to  be 
analyzed.  The  ATE  part  of  the  team  must  be  experienced  enough  to  apply 
judgment  to  fit  the  inevitable  gaps  that  will  appear  in  the  prime  equipment  test 
requirements  definition  during  early  acquisition  phases.  Logisticians  are 
needed  to  ensure  that  ILS  goals  are  met,  and  the  whole  must  be  under  the 
direction  of  the  SYSCOM  to  maintain  program  integrity. 

1.4  PROCEDURE  SUMMARY 

The  ATE  selection  procedure  can  be  simply  summarized  in  the  following  steps: 

(1)  Determine  Test  Requirements 

This  step  requires  knowledge  of  the  prime  equipment  configuration  and 
its  logistic  support  policies. 

(2)  Identify  ATE  Candidates 

Locate  ATEs  in  existing  inventory  which  will  meet  test  requirements. 

If  necessary,  postulate  new  ATE  for  this  purpose. 

(3)  Evaluate  Candidates 

Compare  candidates  on  basis  of  technical  performance,  logistics,  cost, 
and  other  predetermined  evaluation  criteria. 

(4)  Select  the  Optimum  Candidate 

The  ATE  selected  will  provide  the  optimum  match  to  requirements. 
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1.5  ORGANIZATION  OF  THE  REPORT 
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Section  2 describes  ATE  selection  decisions  as  they  relate  to  the  various 

A 

acquisition  phases.  Section  3 describes  the  ATE  decision  process  relative  to 
maintenance  levels,  and  also  contains  a discussion  of  typical  built-in  test 
approaches  since  this  is  an  area  of  increasing  interest  and  application.  Section  4 
covers  specifics  of  the  steps  in  the  ATE  selection  procedure,  and  because 
modeling  can  be  a useful  tool  in  ATE  selection,  a subsection  discusses  types 
and  capabilities  of  logistic- related  models.  Section  5 contains  examples  of 
actual  systems  which  have  utilized  ATE  and  describes  the  process  by  which  the 
particular  ATE  type  was  chosen.  Section  6 contains  appendices  which  define 
ATE  selection  evaluation  factors,  a check  list  of  items  to  be  considered  in  ATE 
selection,  a glossary  of  ATE-related  terms,  and  a comprehensive  discussion 
of  data  banks1  and  their  applicability  to  the  ATE  selection  process. 
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This  outline  is  a preliminary  one  intended  by  NELC  as  a means  of  coordinating 
the  work  of  the  two  contractors.  However,  the  results  of  the  data  bank  task 
(by  Hughes  Aircraft)  are  so  extensive  that  it  is  considered  likely  that  they  will 
constitute  a separate  document  when  NELC  integrates  the  RCA  and  Hughes 
reports. 
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SECTION  2 


SUPPORT  DECISIONS  RELATED  TO  PRIME  EQUIPMENT 
DEVELOPMENT  CYCLE 

2.1  ON-LINE  VS.  OFF-LINE1 


Section  6.3  contains  definitions  which  should  eliminate  the  confusion  often  gen- 
erated by  the  proliferation  of  redundant  and  conflicting  automatic  test  equipment 
terms.  The  most  misunderstood  terms  are  on-line  and  off-line,  and  because 
of  their  relevance  to  the  following  discussion,  a further  attempt  will  be  made  to 
clarify  their  meanings.  The  on-llne/off-line  confusion  arises  because  those 
terms  are  used  to  describe  test  equipment  placement  as  well  as  operational 
mode,  and  the  two  are  not  necessarily  consistent.  Built-in  test  equipment 
(BITE)  is  defined  as  on-line  because  it  is  inherently  connected  to  the  equipment 
it  supports.  However,  some  of  the  tests  performed  by  BITE  may  require  inter- 
rupting normal  operation  of  the  prime  equipment  — that  is,  taking  the  prime 
equipment  "off-line. " BITE,  then,  although  an  on-line  equipment  in  the  sense 
of  location  or  placement,  can  and  usually  does  operate  both  in  on-line  and  off- 
line test  modes. 

A simple  digital  voltmeter  normally  stored  on  a laboratory  shelf  is  off-line  test 
equipment  in  the  placement  sense.  It  can  be  temporarily  connected  to  an  equip- 
ment to  perform  on-line  or  off-line  tests,  depending  on  whether  the  tests  are 
operationally  non- interfering  or  interfering,  respectively.  Permanently  wired 

1This  subsection  was  not  in  the  original  NELC  outline.  Its  addition  moved 
subsequent  sections  up  by  . 1 . 
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into  a prime  equipment,  it  becomes  on-line  equipment,  regardless  of  test  mode. 
The  AN/SSM-5  (TEAMS)  and  the  AEGIS  MK  545  ORTS  are  centralized  test  sys- 
tems which  are  dedicated  to,  or  always  connected  to,  the  equipment  they  support. 
Therefore,  they  are  classified  as  on-line  equipments,  even  though,  unlike  BITE, 
they  can  be  disconnected  or  removed  from  the  prime  equipment  without  disassem- 
bling or  disabling  the  prime  equipment,  and  even  though  some  of  the  tests  they 
perform  require  taking  the  prime  equipment  off-line. 

In  summary,  based  on  clarifying  discussions  with  Navy  engineering  personnel, 
these  are  the  definitions  of  on-line  and  off-line  as  the  terms  will  be  used  in  this 
report.  ON-LINE  TEST  EQUIPMENT  is  test  equipment  which  is  dedicated  and 
permanently  connected  to  the  prime  equipment,  regardless  of  whether  the  test 
equipment  is  built-in  or  separate  from  the  prime  equipment.  OFF-LINE  TEST 
EQUIPMENT  is  test  equipment  which  is  not  installed  as  part  of,  or  permanently 
connected  to,  the  prime  equipment.  ON-LINE  TESTING  is  testing  which  can  be 
performed  while  the  test  equipment  is  in  operational  use,  and  without  interfering 
with  its  operation.  It  may  be  performed  by  on-line  or  off-line  test  equipment. 
OFF-LINE  TESTING  is  testing  which  necessitates  interruption  of,  or  interfer- 
ence with,  prime  equipment  normal  operation.  It,  too,  may  be  performed  by 
on-line  or  off-line  test  equipment. 

2.2  CONCEPTUAL  PHASE 


2.2.1  Introduction 


The  confidence  with  which  test  equipment  decisions  can  be  made  depends  almost 
entirely  on  the  information  available  for  the  equipment  to  be  supported.  Obviously, 
the  supported  equipment  will  be  least  well  defined  during  the  Conceptual  Phase, 


and  only  late  in  the  Validation  Phase  can  modules  and  their  detailed  test  require- 
ments be  identified.  Nevertheless,  basic  test  equipment  decisions  do  need  to  be 
made  during  the  Conceptual  Phase,  and  it  is  clear  that  the  decision  process  will 
require  considerable  judgment  and  experience  in  the  absence  of  firm  and  detailed 
technical  information  during  that  phase.  The  decision  to  use  built-in  test  equip- 
ment (defined  here  as  test  equipment  which  is  an  integral  part  of  the  prime  equip- 
ment) must  be  made  during  the  Conceptual  Phase. 

Deferring  that  decision  to  a later  phase  is  inviting  a costly  design  change  in  the 
prime  equipment  at  that  time.  Somewhat  less  urgent  is  the  decision  to  use 
external  test  equipment.  Even  in  this  case,  however,  there  are  reasons  for 
making  that  decision  during  the  Conceptual  Phase.  The  possible  need  for  test 
equipment  interface  hardware  within  the  prime  equipment  is  one  such  reason. 

And  even  if  no  integral  interface  equipment  is  required,  if  the  test  equipment 
or  test  method  can  be  specified  before  validation,  the  design  can  better  provide 
for  the  necessary  test  access.  While  the  foregoing  statements  can  generally 
be  applied  to  mechanical  as  well  as  electronic  equipment,  test  decisions  for 
mechanical  equipment  which  will  need  relatively  few  sensors  for  test  purposes 
could  be  deferred  to  the  Validation  Phase  with  a lesser  risk  of  major  design 
changes.  An  example  of  mechanical  equipment  for  which  test  decisions  should 
be  made  during  Conceptual  is  a gas  turbine  engine.  A larger  number  of  sensors 
must  be  planned  for  than  are  needed  for  normal  operational  instrumentation, 
particularly  if  gas  path  analysis  is  selected  as  the  means  for  evaluating  engine 
performance. 

Figure  2-1  summarizes  the  types  of  ATE  decision  that  can  be  made  during  the 
Conceptual  Phase,  and  the  format  for  defining  those  decisions.  The  decision 
as  to  whether  to  use  on-line,  off-line,  or  both  classes  of  test  equipment  is 


CONCEPTUAL 


ON-LINE?  OFF-LINE 


made  during  this  period.  Also,  whether  the  on-line  equipment  should  be  built-in 
or  external  should  be  defined,  along  with  the  maintenance  level  at  which  off-line 
test  equipment  will  be  used.  The  results  of  these  decisions  are  summarized  in 
top  level  specifications  for  the  ATE  hardware  and  software.  Impact  on  the  prime 
equipment  anticipated  from  the  ATE  interface  will  be  defined  in  a general  way, 
as  will  the  fault  isolation  level  for  on-line  equipment.  It  will  be  possible  to  be 
more  specific  about  the  fault  isolation  level  for  off-line  test  equipment,  because 
off-line  test  equipment  is  usually  not  hampered  by  the  stimulus,  space,  opera- 
tional non-interference,  and  other  considerations  which  limit  the  fault  isolation 
capability  of  on-line  test  equipment. 

2.2.2  Meeting  Availability  Goals 

Availability  of  the  prime  equipment  will  be  specified  to  be  consistent  with  mis- 

MTBF 

sion  requirements.  Availability  = Improving  MTBF  in  order 
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to  increase  availability  is  not  a function  of  test  equipment  selection,  but  is  deter- 
mined by  prime  equipment  complexity  and  component  state  of  art.  However, 
MTTR  and  thus  availability,  can  be  improved  by  the  selection  of  test  equipment 
which  will  reduce  the  portion  of  repair  time  required  for  diagnosis  of  the  faulty 
component  or  assembly,  and  for  retest  after  repair.  Where  availability  goals 
are  difficult  to  meet,  consideration  should  be  given  to  the  use  of  on-line  test 
equipment  in  order  to  minimize  diagnostic  time.  The  designer  has  the  choice 
of  specifying  built-in  test  equipment  or  an  external  test  set  which  is  always 
connected  to  the  prime  equipment.  In  either  case,  since  diagnostic  time  is 
only  a portion  of  MTTR,  the  prime  equipment  must  be  designed  to  minimize 
access  time  for  parts  replacement.  Also,  the  test  equipment  availability 
should  exceed  that  of  the  prime  equipment  (ideally,  10  to  1),  and  similar  design 
practices  should  be  applied  to  achieve  that  goal.  With  on-line  test  equipment 
complexity  usually  less  than  10  - 15  percent  of  the  prime,  a higher  MTBF  than 
the  prime  is  assured  at  least  on  a parts  count  basis,  but  self-test  is  still  nec- 
essary to  sense  test  equipment  failures  which  can  cause  false  goes  and  no-goes 
in  the  test  results,  and  to  minimize  the  test  equipment  MTTR. 

2.2.3  The  Role  of  the  Maintenance  Engineering  Analysis  (MEA) 

The  MEA  is  initially  made  during  the  Conceptual  Phase.  It  will  necessarily  be 
incomplete  at  that  point  because  of  the  lack  of  technical  detail  which  can  be 
made  available.  The  MEA  is  therefore  updated  through  subsequent  prime 
equipment  acquisition  phases.  Maintenance  Engineering  Analysis  (MEA)  forms 
are  usually  used  to  specify  test  equipment  requirements  and  thus  are  normally 
not  useful  as  raw  inputs  to  an  ATE  selection  process.  However,  where  logistic 
funding  permits,  and  often  this  is  only  the  case  on  major  weapon  system  pro- 
curements, the  MEA  process  can  be  expanded  during  validation  to  specify 
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detailed  test  requirements  at  the  stimulus  and  measurement  level  for  each 
module,  thereby  enabling  test  equipment  selection  to  be  made  with  a high  degree 
of  confidence. 

During  the  Conceptual  Phase,  the  MEA  can  only  assist  in  making  broad  test 
decisions,  such  as,  for  example,  whether  an  on-line  or  off-line  test  equipment 
approach  should  be  chosen  to  meet  availability  requirements,  and  selecting  the 
equipment  level  to  which  fault  isolation  should  function. 

2.3  VALIDATION  PHASE 


During  the  Validation  Phase,  detailed  design  decisions  can  be  made  for  test 
equipment  options  previously  selected  during  Conceptual.  To  decide  to  incor- 
porate built-in  test  after  validation  has  been  contracted  for  could  result  in 
costly  design  changes.  However,  the  final  selection  of  off-line  test  equipment 
could  be  made  as  technical  details  of  the  prime  equipment  unfold  to  enable  a 
fine-grain  analysis  of  test  requirements  to  be  made  — possibly  via  the  MEA 
procedure.  This  is  the  phase  during  which  a module  tester  could  be  selected. 
Figure  2-2  summarizes  the  ATE  decisions  of  the  Validation  Phase,  which  are 
largely  those  which  expand  in  detail  on  major  decisions  previously  made. 
Results  consist  of  detailed  ATE  hardware  and  software  specifications,  and 
either  selection  of  existing  test  equipment,  or  specification  and  start  of  design 
for  new  test  equipment.  Details  of  the  prime  equipment  interface  will  be 
developed  as  the  prime  equipment  design  becomes  better  defined.  The  func- 
tioning and  specifics  of  fault  isolation  will  also  become  definable,  with  partic- 
ular impact  on  on-line  test  equipment.  At  this  stage  the  need  for  compromises 
on  on-line  performance  will  become  apparent,  and  it  may  be  found  that  "fault 
isolation  to  the  module  level"  may  in  some  instances  have  to  mean  fault  isolation 
to  a group  of  modules  rather  than  always  to  an  individual  module.  Packaging 
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VALIDATION 


Figure  2-2.  Validation  Phase  ATE  Decisions 

and  circuit  partitioning  of  the  prime  equipment  form  one  of  the  major  constraints 
on  the  fault  isolation  capability  of  on-line  test  equipment.  Maximizing  packaging 
efficiency  and  standardization  leads  to  the  design  of  modules  containing  many 
identical  circuit  elements,  such  as  all  flip-flops  or  all  gates,  to  be  distributed 
over  a number  of  subsystem  functions.  The  many  input  and  output  lines  of  these 
modules  make  them  difficult  to  fault  isolate  with  on-line  test  equipment.  Inevitably, 
compromises  between  prime  and  test  equipment  design  goals  will  be  necessary, 
and  these  are  best  resolved  where  the  same  engineering  management  directs  the 
prime  and  the  on-line  test  equipment  design  tasks. 

2.4  PRODUCTION  PHASE  - HUGHES  AIRCRAFT  CORPORATION 


2.5  DEPLOYMENT/OPERATIONAL  PHASE  - HUGHES  AIRCRAFT  CORPORATION 


SECTION  3 


SUPPORT  ALTERNATIVES  BY  MAINTENANCE  LEVEL 
3.1  ORGANIZATIONAL 

The  selection  of  support  alternatives  for  the  organizational  level  requires  crit- 
ical technical  decisions  to  be  made  early  in  the  acquisition  program  before 
technical  details  have  been  developed.  Organizational  test  equipment  is  more 
tightly  bound  by  physical,  cost,  and  manpower  constraints  than  is  test  equip- 
ment intended  for  other  maintenance  levels.  On-line  test  equipment,  whether 
built-in  or  external,  is  a strong  contender  at  this  level,  and  the  added  cost  per 
installation  multiplied  by  the  number  of  installations  requires  consideration  of 
MTTR  reducing  methods  which  do  not  rely  entirely  on  test  equipment.  Figure 
3-1  illustrates  the  range  of  generally  acceptable  BITE  complexity  expressed  as 
a percentage  of  the  prime  equipment.  Two  percent  is  the  approximate  minimum, 
although  it  could  run  lower  for  a simple  detector  to  indicate  the  presence  of 
power.  More  significant  is  the  maximum,  which  is  set  at  10-15  percent  to  en- 
sure an  MTBF  significantly  higher  than  the  prime  equipment,  and  to  avoid  an 
intolerable  increase  in  acquisition  cost.  Where  BITE  may  require  a greater 
percentage  of  the  prime,  self-healing  and  adaptive  circuitry  become  serious 
contenders,  although  as  will  be  discussed,  these  methods  can  be  costly  and  do 
not  necessarily  eliminate  the  need  for  organizational  test  equipment. 


Design  approaches  will  be  briefly  discussed  in  this  section  which  can  replace 
or  augment  organizational  test  equipment. 


Figure  3-1.  BITE  Relative  to  the  UUT 
3.1.1  Built-in  Self-Healing 

Built-in  self-healing  is  a design  technique  which  can  reduce  and  perhaps  elim- 
inate built-in  test  equipment.  It  is  beyond  the  scope  of  this  report  to  attempt 
detailed  coverage  of  this  technique.  However,  it  is  discussed  here  because  it 
must  be  considered  as  an  alternative  to  built-in  test  equipment.  Active  self- 
healing  would  detect  the  presence  of  a fault  and  automatically  switch  over  to  a 
standby  circuit.  Since  active  self-healing  must  contain  a fault-detection  mech- 
anism, it  can  provide  a BITE  function  by  externally  displaying  the  malfunction 
detected. 

Passive  self-healing  is  achieved  by  redundant  circuitry  which  automatically  as- 
sumes the  function  that  was  performed  by  a failed  component,  without  the  need 
for  fault  detection.  Figure  3-2  illustrates  the  technique  as  applied  to  a simple 
diode  path.  Here  four  diodes  are  connected  in  what  is  sometimes  known  as  a 
"hammock”  circuit.  It  can  be  seen  that  a normal  diode  path  will  continue  to 


REDUNDANCY  OF  COMPONENTS 


DIODE  PATH  CONTINUES  TO  EXIST  WITH: 

1)  ONE  DIODE  SHORTED  IN  EACH  LEG,  OR 

2)  BOTH  DIODES  IN  ONE  LEG  OPENED,  OR 

3)  i)  FLUS  ONE  DIODE  SHORTED  IN  OTHER  LEG. 


Figure  3-2.  Component  Redundancy 

exist  for  a variety  of  combined  failures.  Figure  3-3  shows  two  dc  power  supplies 
isolated  from  each  other  by  diodes  but  feeding  the  same  load.  If  both  supplies 
provide  the  same  output  voltage  they  will  equally  share  the  load.  If  one  goes 
down,  the  survivor  will  supply  the  entire  load.  Each  supply  would  have  to  be 
capable  of  handling  the  entire  load.  The  degree  of  redundancy  can  be  increased 
to  improve  operational  reliability,  but,  clearly,  at  a further  increase  in  cost 
and  complexity.  The  danger  in  passive  self-healing  is  that  there  is  no  way  of 
knowing  whether  or  not  the  equipment  is  working  on  its  last  reserve  component 
or  assembly.  Auxiliary  BITE  devices  would  be  needed  to  detect  a failure  of 
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Figure  3-3.  Assembly  Redundancy 


one  of  the  power  supplies  in  figure  3-3.  However,  detecting  failures  in  re- 
dundant components,  such  as  the  diodes  of  figure  3-2,  would  not  be  as  straight- 
forward. To  avoid  the  need  to  probe  the  diodes  individually,  precise  voltage 
readings  across  the  entire  network  would  be  needed,  with  the  network  replace- 
able as  one  unit.  Simple  examples  have  been  presented  here.  Much  more 
complex  variations  of  self-healing  will  be  apparent  to  designers  of  digital  equip- 
ment. However,  the  problem  of  detecting  failures  when  the  entire  system  con- 
tinues to  operate  normally  remains  a serious  one. 


Voting  circuits  are  another  form  of  self-healing,  and  they  may  or  may  not  con- 
tain an  inherent  failure  detection  mechanism.  Figure  3-4  is  an  example  of 
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Figure  3-4.  Voting  Logic  Application 
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voting  logic  dictated  by  flight  safety  considerations.  Shown  is  a method  for 
using  three  radar  altimeters  in  an  automatic  landing  system. 

Altimeter  A is  the  primary  altimeter,  and  its  output  will  be  used  if  it  is  in 
agreement  with  either  B or  C.  Altimeter  B is  the  backup  to  A,  and  C is  used 
only  for  voting  purposes.  This  is  a simplified  example.  Obviously,  a thresh- 
hold  of  agreement  would  need  to  be  established,  and  also  the  logic  would  have 
to  recognize  ''agreement"  due  to  two  or  more  sets  being  turned  off. 

3.1.2  BITE  On-Line  Fault  Isolation  to  the  Replaceable  Module 

The  ideal  in  ATE  operation  is  to  fault  isolate  to  the  replaceable  module  level, 
using  built-in  test  equipment  (BITE),  while  the  prime  system  is  on-line  (oper- 
ational). To  achieve  this  goal  requires  either  that  normal  operation  provide  the 
necessary  stimulus,  or  that  it  can  be  applied  by  the  BITE  without  interfering 
with  the  module's  normal  functioning.  Oscillator,  clock,  and  power  supply 
modules  are  self-stimulating  and  can  be  fault- isolated  by  output  measurement. 
Amplifiers  through  which  a signal  of  known  characteristics  flows  during  normal 
operation  can  also  be  fault- isolated  on-line  by  simple  measurement.  Pulsed 
oscillators  or  generators  where  (he  measurement  can  bo  synchronized  with  the 
trigger  also  lend  themselves  to  on-line  fault  isolation.  Mechanical  devices  are 
often  naturally  self-stimulating.  Pressure  systems,  engines  and  other  rotating 
machinery  are  fault  isolatable  on-line  with  a much  lower  sensor  or  test  point 
density  than  is  usually  encountered  with  electronic  equipment.  Radio  and  radar 
receiver  modules  present  special  problems  because  the  wide  range  of  normally 
encountered  signal  types  and  intensities  are  too  unpredictable  to  be  suitable  as 
test  stimuli.  Modules  of  a radar  receiver,  however,  could  be  externally  stim- 
ulated b>  BITE  during  the  transmission  part  of  the  radar  cycle,  although  that 
would  not  be  on-line  testing  in  the  purest  sense  of  the  term.  Digital  modules 
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can  be  fault- isolated  on-line,  similarly  to  the  case  of  the  radar  receiver,  by 
stimulating  them  and  measuring  response  between  operational  windows.  This 
is,  in  effect  a time-sharing  technique,  where  test  modules  in  the  operating 
software  feed  test  patterns  and  evaluate  response  between  normal  computational 
pulses. 

As  mentioned  previously,  packaging  efficiency  and  standardization  are  often 
the  enemies  of  on-line  fault  isolation,  because  they  tend  to  complicate  the  stim- 
ulus and  test  logic  required  to  fault  isolate  to  a single  module.  The  ideal  mod- 
ule for  fault  isolation  is  one  with  a single  input  and  output.  If  a module  contains 
an  IF  amplifier  along  with  stray  components  from  other  functions  for  which 
space  happened  to  be  available  on  the  IF  module,  then  fault- isolating  the  mod- 
ule will  obviously  be  considerably  complicated  by  the  need  to  test  the  sub- 
functions or  sub- sub- functions  represented  by  the  odd  components.  Digital 
modules  are  often  standardized  for  good  logistic  reasons.  But  where  a standard 
module  can  contain  dozens  of  identical  logic  gates  for  external  connection  to 
many  logic  circuits,  there  is  an  obvious  problem  in  isolating  failures  to  that 
module,  and  the  continuing  development  of  computer  programs  for  automatic 
generation  of  digital  test  patterns  represents  the  most  likely  solution  to  the 
problem . 

Regardless  of  whether  fault- isolation  is  to  be  accomplished  on-line  or  off-line, 
its  achievement  requires  a team  effort  involving  the  circuit  design  engineer, 
the  packaging  engineer,  and  the  test  engineer.  Circuit  partitioning  is  the  most 
important  factor  in  the  success  of  BITE. 
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3.1.3  BITE  Off-Line  Testing  to  the  Replaceable  Module 


With  the  inhibition  of  operational  non-interference  removed,  it  becomes  con- 
siderably simpler  to  isolate  faults  to  a replaceable  module  with  BITE.  However, 
the  practical  limitations  on  space  for  stimulus  generators  and  access  to  input 
terminals  keeps  BITE  from  achieving  the  test  capability  of  off-line  (external) 
test  equipment.  In  the  future,  microcircuit  developments  should  alleviate  the 
problem  and  increase  the  application  of  BITE  techniques. 

Figure  3-5  is  a simplified  diagram  of  a BITE  approach  developed  for  a commu- 
nication transceiver  which  achieves  fault  isolation  to  the  replaceable  module, 

# 

using  a combination  of  on-line  and  off-line  test  techniques.  Only  the  receiver 
portion  is  shown  to  illustrate  the  technique  as  it  is  applied  by  RCA  to  the  Mk 
LC  HF  Transceiver,  a successor  to  the  AN/ARC-161  HF  Transceiver  which  is 
too  new  to  have  been  assigned  an  AN/ARC-number.  Frequency  generator  mod- 
ules of  the  frequency  synthesizer,  and  power  supply  modules  are  basically  self- 
stimulating,  so  they  are  continuously  monitored  on-line,  and  a single  fault 
light  indicates  failure  of  any  one  of  those  modules,  while  a numerical  readout 
indicates  the  particular  faulty  module.  If  it  is  desired  to  run  a routine  check, 
the  operator  presses  a button  and  briefly  interrupts  normal  operation  to  initiate 
a complete  self-test.  A signal  generated  within  the  frequency  synthesizer  is 
switched  to  the  antenna  terminal  in  place  of  the  antenna,  and  an  automatic  se- 
quencer steps  through  each  module  in  turn,  starting  at  the  front  end  and  pro- 
ceeding toward  the  audio  output,  while  testing  for  module  failure  by  evaluating 
the  signal  at  each  inter-module  junction.  When  a failure  is  detected,  the  se- 
quence stops  and  an  indicator  displays  the  number  of  the  module  to  be  replaced. 
The  success  of  this  method  resulted  from  a coordinated  design  effort  whereby 
circuit  partitioning  and  packaging  were  directed  toward  the  goal  of  completely 
automatic  built-in  module  fault- isolation.  The  percentage  increase  in  system 
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complexity  through  the  use  of  BITE  was  well  under  five  percent  in  this  case, 
thanks  to  the  ingenious  minimization  of  stimulus.  The  only  drawback  in  this 
approach  would  seem  to  be  that,  in  case  of  multiple  failures,  only  one  faulty 
module  at  a time  could  be  detected.  However,  the  occurrence  of  multiple  fail- 
ures is  low  in  probability,  and  to  generate  a separate  and  unique  stimulus  for 
each  module  in  order  to  allow  for  that  remote  possibility,  would  have  enormously 
and  unacceptably  increased  the  BITE  complexity.  Although  the  transmitter  unit 
delivers  one  kW  output,  an  18  watt  internal  dummy  load  is  sufficient  for  test 
purposes  because  of  the  limited  test  duty  cycle. 


BITE  OPERATION  CHARACTERISTICS 


LIGHT  INDICATES  FAULT  IN  ANY  ONE  SET  OF  MODULES  AUTOMATICALLY  TESTED 

DEVICE  PROVIDING  CONTINUOUS  OUTPUT  ON-LINE  AS  A GROUP 


OPERATOP  INITIATES  MODULE  FAULT  TEST 

FAULT  DISPLAY  INDICATES  NUMBER  OF 
FIRST  MODULE  IN  CHAIN  TO  FAIL 


OPERATOR  INITIATED  OFF-LINE  FAULT  ISOLATION  TO 
MODULE  LFVEL 

END-TO-END  PERFORMANCE  TESTED  OFF-LINE 


Figure  3-5.  Mk  LC  Radio  Receiver  BITE 


3-8 


3.1.4  Off-Line  Testing  - General 


Organizational  off-line  testing  can  be  performed  by  off-line  test  equipment  or 
it  can  consist  of  end-to-end  or  sub-system  testing  by  BITE  or  by  other  on-line 
test  equipment.  Module  fault  isolation  can  conceivably  be  performed  off-line 
while  the  assembly  which  contains  the  modules  remains  in  its  normal  operating 
location.  It  is  more  likely,  however,  that  the  assembly  will  be  removed  and 
taken  elsewhere  to  the  location  of  the  automatic  test  facility.  Fault  isolation  in 
place  can  be  done  by  probing  with  manual  test  equipment,  or  by  the  use  of  por- 
table automatic  test  equipment.  In  any  case,  testing  can  be  facilitated  by  pro- 
viding a test  connector  (as  is  done  for  VAST)  to  enable  access  to  test  points 
without  disassembling  the  equipment.  Where  test  points  required  for  fault 
isolation  to  a module  are  critical  with  regard  to  impedance  or  cross-talk,  it 
may  not  be  possible  to  wire  them  to  an  external  connector  without  degrading 
performance.  One  possible  solution  would  be  to  place  an  isolation  circuit  be- 
tween test  point  and  test  connector.  The  alternative  is  disassembly  and  probing 
of  the  circuit,  which  requires  dexterity  and  care  on  the  part  of  the  operator  to 
avoid  damage.  As  with  BITE,  circuits  should  be  partitioned  and  modules  laid 
out  to  fficili*n»r>  fault  isolation 

OH-line  fault  isolation  lo  a module  level  can  be  a two  step  operation  where  it  is 
not  practicable  to  bring  out  the  test  points  necessary  to  locate  the  faulty  module 
on  the  first  pass.  The  first  step  would  locate  the  malfunction  to  a group  of 
modules  (obviously,  the  smaller  the  group,  the  better),  and  the  modules  in  that 
groiq?  would  be  removed  from  the  parent  assembly  and  individually  tested  man- 
ually or  by  an  automatic  module  tester  to  pin-point  the  defective  one.  The 
techniques  for  off-line  end-to-end  or  sub-system  testing  are  similar  whether 
BITE,  other  on-line  test  equipment,  or  off-line  test  equipment  is  used.  BITE 


is  handicapped  relative  to  the  latter  two  by  the  constraints  that  go  with  being 
physically  part  of  the  prime  equipment.  However,  BITE  has  an  advantage  in 
closer  access  to  internal  test  points,  which  could  reduce  the  need  for  special 
test  point  isolation  circuitry  relative  to  requirements  with  external  test  equip- 
ment. Examples  will  be  given  of  organizational  test  equipment  which  performs 
tests  at  higher  levels  than  the  replaceable  module.  NELC  has  developed  a 
receiver  noise  figure  measuring  device  which  could  be  incorporated  in  a re- 
ceiver as  BITE  to  perform  overall  tests,  or  which  could  be  used  as  off-line 
organizational  test  equipment.  Figure  3-6  shows  the  device  in  a BITE  appli- 
cation. The  receiver  is  taken  off-line,  and  a noise  generator  is  used  as  stim- 
ulus, obviating  the  need  for  tuning.  The  noise  figure  reading  developed  is  a 
measure  of  the  condition  and  circuit  alignment  of  the  entire  receiver. 


OUTPUT  DEVICES 


BITE  OPERATION 


CHARACTERISTICS 


OPERATOR  SETS  GAIN/SQUEICH  CONTROLS  AS  REQUIRED 

OPERATOR  SETS  BANDWIDTH  CONTROL  AS  REQUIRED 

OPERATOR  SETS  BAND/FREQUENCY  OF  INTEREST 

OPERATOR  DISABLES  A GC 

OPERATOR  DISABLES  NOISE  LIMITER 

OPERATOR  INITIATES  TEST 

DISPLAY  INDICATES  NOISE  FIGURE 


OPERATOR  INITIATED 

NORMAL  OPERATION  MOMENTARILY 
INTERRUPTED  (OFF-LINE) 

END-TO-END  PERFORMANCE  TESTED 


Figure  3-6.  NELC  Receiver  Noise  Figure  Tester-Shown  Adapted  as  BITE 
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Figure  3-7  illustrates  off-line  end-to-end  test  of  an  AN/APN-155  altimeter 
using  BITE  hardware.  The  operator  depresses  a test  button  which  switches  a 
calibrated  delay  cable  in  place  of  the  antennas.  The  indicator  will  then  read 
the  test  altitude  if  the  entire  set  is  functioning  properly.  No  faulty  isolation 
is  provided. 


ALTITUDE 

INDICATOR 


TERRAIN 

BITE  OPERATION 

OPERATOR  PUSHES  TEST  BUTTON  ON  INDICATOR 
CALIBRATED  DELAY  CABLE  SWITCHED  IN  PLACE  OF  ANTENNAS 
INDICATOR  DRIVES  TO  CALIBRATE  ALTITUDE 
OPERATOR  READS  AND  INTERPRETS  RESULTS 

CHARACTERISTICS 

OPERATOR  INITIATED 

NORMAL  OPERATION  MOMENTARILY  INTERRUPTED  (OFF-LINE) 

END-TO-END  PERFORMANCE  TESTED  (LESS  ANTENNAS) 

Figure  3-7.  AN/APN-155  Radar  Altimeter  BITE 

The  AEGIS  Mk  545  Operational  Readiness  Test  System  (ORTS)  is  depicted  in 
simplified  block  diagram  form  in  Figure  3-8.  This  is  an  example  of  an  on-line 
(always  connected  to  the  prime  equipment)  centralized  automatic  test  system. 

It  performs  on-line  and  off-line  tests  from  overall  system  operability  down,  in 
some  cases,  to  the  module  level.  Practical  limitations  on  the  partitioning  of 


OPERATION  AND  TEST  CONTROL 


BITE  OPERATION 


AUTOMATICALLY  AND  CONTINUOUSLY  TESTS  OPERABILITY  OF  PART  OF 
SYSTEM  ON-LINE 

PERFORMS  OFF-LINE  OPERABILITY  AND  FAULT  ISOLATION  TESTS  UPON 
OPERATOR  COMMAND 


CHARACTERISTICS 


SHARES  AEGIS  CENTRAL  PROCESSING  COMPLEX 

SYNCHRONOUS  AND  NON-SYNCHRONOUS  TESTING  OF  DIGITAL 
SUBSYSTEMS  ACCOMPLISHED  THROUGH  CENTRAL  PROCESSOR 

MULTIPLEX  SYSTEM  SIMPLIFIES  DATA  GATHERING  EQUIPMENT 

COMBINED  ON-LINE  AND  OFF-LINE  METHODS  USED 

OPERATOR  INITIATES  OFF-LINE  OPERATION 


Figure  3-8.  Aegis  Mk  545  Operational  Readiness  Test  System  (ORTS) 

I lip  prime  equipment,  which  wore  largely  dictated  by  logistic  considerations, 
resull  in  liiuli  Isolation  HnmcIlmcH  nloppiug  nl  I lit'  level  of  groups  ot  modules. 
The  intent  is  to  use  ORTS  with  a module  lester  for  those  eases,  providing  two 
step  module  fault  isolation  where  necessary.  ORTS  is  computer-controlled 
and  shares  the  data  processing  sub- system  of  the  AEGIS  weapon  system  which 
it  supports.  The  only  portions  of  ORTS  which  are  internal  to  the  supported 
equipment  are  the  interface  assemblies  through  which  test  data  are  multiplexed 
to  the  ORTS  control  consoles  and  computers.  Analog  test  signals  are  all  con- 
ditioned and  normalized  to  a dc  range,  similarly  to  guidelines  established  in 
MIL-STD-1326.  Serial  and  parallel  digital  signals  are  acquired  through  the 


3-12 


data  acquisition  interface,  either  synchronously  or  non-synchronously.  Test 
points  are  automatically  interrogated  in  a sequence  determined  by  the  test  soft- 
ware, or  by  manual  override  of  the  operator . Responses  are  received  on  the 
same  multiplex  data  cable,  and  the  shared  control  processor  operates  on  the 
data  and  displays  results  at  control  consoles.  Although  an  elaborate  system, 
ORTS  services  a highly  sophisticated  weapon  system,  and  the  added  complexity 
due  to  ORTS  is  estimated  at  under  the  10-15  percent  maximum  established  for 
BITE.  ORTS  is  an  example  of  organizational  equipment  which  is  comprehensive 
enough  to  eliminate  all  additional  test  equipment  except  for  a module  tester. 

3.2  INTERMEDIATE  LEVEL  EQUIPMENT  - Hughes  Aircraft  Corporation 


3.3  DEPOT  LEVEL  EQUIPMENT  - Hughes  Aircraft  Corporation 


SECTION  4 


SELECTION  PROCEDURES 

The  selection  procedures  described  herein  are  accompanied  by  background 
material  to  familiarize  the  users  with  logistic  considerations,  generic  ATE 
types,  and  economic  analysis  methods.  The  procedures  were  developed  for 
SYSCOM  users  and  others  whose  specialty  is  in  the  prime  equipment  for  which 
they  are  directly  responsible,  and  whose  ATE  task  consists  of  seeing  that  the 
proper  resources  are  directed  toward  the  selection  of  the  most  cost-beneficial 
ATE.  Since  the  ATE  selection  procedures  are  essentially  intended  as  project 
and  technical  management  tools,  the  background  material  is  held  to  a level 
consistent  with  that  intent.  No  attempt  was  made  to  duplicate  the  extensive 
library  which  is  available  elsewhere  to  the  specialist  in  logistics,  ATE,  and 
economic  analysis. 

4.1  IMPACT  OF  THE  LOGISTIC  SUPPORT  CONCEPT 

Alternative  maintenance  concepts  will  bo  synthesized  by  the  SYSCOM  as  part 
of  Integrated  Logistics  Support  concept  formulation.  Maintenance  alterna- 
tives will  be  directed  toward  optimizing  acquisition  cost,  life  cycle  cost, 
manning,  skill  levels,  and  equipment  availability.  Alternatives  will  be  based 
on  defining  the  spares  policy,  the  level  of  repair  and  the  repair/discard 
policies  at  each  maintenance  level.  Maintenance  alternatives  are  likely  to  be 
straightforward  and  consistent  for  individual  equipments  or  systems.  Where 
an  extensive  installation  of  heterogeneous  equipment  is  being  planned,  as  for 
an  aircraft  or  ship  design,  maintenance  policies  can  become  more  complicated. 
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For  example , while  it  may  be  economically  acceptable  to  discard  defective 
modules  on  one  equipment  because  of  relatively  low  replacement  cost,  such  a 
policy  could  be  prohibitively  expensive  on  another  equipment  which  uses  more 
costly  modules.  Furthermore,  many  equipments  still  in  inventory  are  not 
modular  in  construction  (e.  g. , R-390A/URR),  and  if  such  equipments  are  to 
be  repaired  at  all  on  a ship,  it  will  have  to  be  to  the  component  level,  regard- 
less of  the  level  of  repair  specified  for  other  equipment  on  the  ship. 

The  fact  that  a system  or  equipment  may  already  be  operational  under  a pre- 
viously established  support  concept  does  not  assure  that  the  same  support 
concept  will  be  satisfactory  for  a new  and  different  application.  Different 
maintenance  concepts  are  possible  for  airborne,  shipboard,  and  land  usage  (as 
by  the  Marine  Corps)  of  the  same  equipment. 

The  level  of  repair  and  availability  requirements  will,  of  all  logistic  considera- 
tions, have  the  greatest  impact  on  ATE  selection.  The  ATE  will  have  to  provide 
a fault  isolation  level  which  is  consistent  with  the  specified  level  of  repair. 

Since  the  level  of  repair  policy  does  not  necessarily  limit  the  means  for 
accomplishing  the  required  fault  isolation  level,  the  ATE  evaluator  can  consider 
both  on-line  and  off-line  options.  However,  a tight  availability  requirement 
could  dictate  an  on-line  ATE  configuration  for  achieving  the  desired  level  of 
repair. 


The  processes  of  selecting  and  formulating  the  logistic  support  concept  can  be 
iterative,  as  concepts  are  successively  modified  for  best  match  with  technical 
feasibility  and  cost  of  the  ATE  and  the  prime  equipment.  The  need  for  iteration 
is  more  apt  to  be  evident  with  BITE  than  with  off-line  or  external  on-line  ATE, 
because  of  tighter  technological,  space,  and  cost  restraints. 
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4.2  RELATIONSHIP  TO  THE  ACQUISITION  PROCESS 


As  touched  on  in  Section  2 in  another  context,  and  as  will  become  more  evident 
later,  the  timeliness  of  ATE  decisions  is  a function  of  the  type  of  ATE  being 
considered  and  the  point  in  the  acquisition  process  at  which  deliberations  are 
in  process.  Generally,  on-line  equipment  must  be  selected  earlier  in  the 
acquisition  process  than  off-line  equipment,  with  BITE  being  the  most  critical 
on-line  approach  with  regard  to  timing.  Since  BITE  is  part  of  the  prime 
equipment,  decisions  relative  to  its  use  and  functions  must  be  made  during  the 
Conceptual  Phase.  If  decisions  are  deferred  until  validation,  potentially  costly 
design  changes  may  be  called  for.  External  on-line  test  equipment  should  also 
be  selected  during  the  Conceptual  Phase,  although  the  risk  incurred  in  waiting 
until  Validation  will  be  less  than  in  the  case  of  BITE,  and  will  depend  upon  the 
amount  of  interface  equipment  which  will  have  to  be  added  integrally  to  the 
prime  equipment  design.  The  selection  of  off-line  test  equipment  could  be  (and 
often  is)  deferred  until  prime  equipment  production  or  beyond.  However,  it  is 
much  more  desirable  to  select  the  ATE  no  later  than  Validation,  because  the 
designer  can  realize  in  advance  the  need  to  configure  the  prime  equipment  to 
facilitate  test  point  accessibility.  He  can  also  partition  circuitry  for  easier 
fault  isolation,  with  significant  benefits  in  testability,  test  time,  and  ATE 
adapter  and  software  costs,  if  he  is  made  aware  of  the  ATE  being  contemplated 
while  he  is  designing  the  prime  equipm  ent. 

4.3  THE  GENERAL  PROCEDURE 

4.3.1  Introduction 


The  general  procedure  for  evaluating  and  selecting  ATE  systems  is  keyed  to  the 
definition  of  the  prime  equipment  logistic  support  concept.  The  logistic  support 


concept  matures  in  parallel  with  the  prime  equipment  acquisition  concept,  as 
MEAs  are  updated  to  reflect  the  increasing  level  of  available  detail  concerning 
configuration  of  the  prime  equipment.  As  stated  previously,  the  ATE  selection 
procedures  were  developed  to  assist  SYSCOMS  in  maxiaging  the  ATE  portion  of 
prime  equipment  programs,  which  involves  directing  and  coordinating  the  work 
of  specialists  assigned  the  details  of  that  task.  ATE  specialists,  if  not  already 
resident  in  the  SYSCOM,  are  envisaged  as  being  enlisted  from  other  Navy 
organizations  or  from  private  industry.  A Test  Equipment  Support  Office 
(TETSO)  has  been  created  at  NELC  as  a source  of  ATE  expertise,  and  TETSO 
engineers  could  either  be  assigned  full-time  to  the  respective  SYSCOM  if  the 
task  warranted,  or  could  provide  consultation  on  an  ad  hoc  basis.  The  risks  of 
implementing  an  ATE  selection  procedure  without  the  assistance  of  ATE 
specialists  are  very  great.  Over-specifying  of  ATE  performance  can  arise  from 
unfamiliarity  with  state-of-art  limitations.  Under- specifying  to  avoid  the 
first  risk  can  result  in  a less  than  optimum  support  system.  Early  in  the 
acquisition  process,  where  technical  details  are  sparse  on  both  the  prime  and 
the  candidate  ATE  system,  prime  equipment  and  ATE  specialists  must  work 
very  closely.  Later  on,  the  need  for  the  prime  equipment  specialist  will  still 
be  required  to  provide  an  informal  evaluation  of  the  firm  candidates.  Later  in 
the  report,  modeling  aids  are  suggested  to  augment  human  judgment  in  the 
selection  process. 

4.3.2  Systems  in  Development 

For  systems  still  in  development,  or  contemplated  to  be  in  development,  entry 
to  the  procedure  is  at  entry  point  1 in  Figure  4-1.  Details  of  equipment  to  be 
supported  would  not  be  firm.  However,  equipment  could  be  described  in  terms 
of  the  numbers  to  be  deployed,  where  they  will  be  used,  and  their  function 
(e.g. , shipboard  HF  communications).  During  preliminary  design,  reliability 


ENTRY 

POINTS  W 


Figure  4-1.  Support  and  Test  Equipment  Evaluation  Procedure 


data  and  equipment  configuration  in  terms  of  module  quantity  could  be  synthe- 
sized by  similarities  to  existing  equipment.  The  maintenance  workload  is  a 
function  of  MTBF's,  MTTR's,  requirements  for  system  availability,  etc.  From 
these  requirements  the  procedure  should  provide  guidance  in  synthesizing 
alternate  support  concepts  such  as  built-in  test,  module  discard,  intermediate 
level  module  repair,  etc.  The  test  equipment  alternatives  would  at  this  point 
be  synthesized  and  described  generically,  that  is,  general  purpose  ATE, 
module  tester,  Go/No-Go  BITE,  etc.  Some  alternatives  would  exist  in  combin- 
nations  such  as,  for  example,  built-in  test  for  overall  operational  monitoring, 


and  off-line  testing  for  module  fault  isolation.  A support  concept  would  be 
selected  from  evaluation  of  alternatives  on  the  basis  of  operational  availability 
(Ao),  hardware  and  manning  costs. 

4.3.3  Support  Concept  Defined 

At  entry  point  2 , the  support  concept  has  already  been  defined  either  as  a 
result  of  the  previous  steps , or  through  other  means  for  defining  the  mainten- 
ance policy.  The  next  step  is  to  define  in  detail  the  technical  requirements 
for  testing  the  equipment  to  be  supported.  For  all  levels  of  test  required  by  the 
support  concept,  down  to  the  lowest  level  of  fault  isolation,  test  requirements 
must  be  defined  in  terms  of  switching,  stimulus,  and  measurement  capabilities. 
Included  would  be  details  on  accuracies,  programmable  increments,  waveforms, 
spectral  purity,  etc.  Where  design  status  is  insufficiently  advanced,  or 
schedule  pressures  do  not  permit,  details  may  be  lacking,  and  the  burden  will 
fall  on  the  ATE  specialist  to  fill  in  the  gaps  based  on  judgment  and  experience. 
The  detail  and  format  for  describing  test  requirements  would  have  to  be 
similar  to  those  used  to  describe  existing  test  equipment  alternatives  in  a data 
bank  expected  to  be  created  for  that  purpose.  Test  requirements  would  be 
compared  to  ATE  capabilities  in  the  data  bank,  and  test  equipment  alternatives 
would  be  identified  which  meet  test  requirements.  If  the  ATE  data  bank  does 
not  provide  a satisfactory  match,  one  or  more  new  ATEs  must  be  specified  and 
evaluated. 

4.3.4  ATE  Candidates  Identified 

At  entry  point  3,  ATE  candidates  have  been  selected  through  previous  steps 
of  the  procedure,  or  through  some  other  process.  The  remainder  of  the 
procedure  is  concerned  with  evaluating  the  alternatives  - essentially  on  the 
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basis  of  economic  factors  - and  selecting  the  optimum  configuration.  Acquisi- 
tion cost,  manning  cost  impact,  program  risk,  and  delivery  are  some  of  the 
factors  to  be  considered.  Although  it  is  anticipated  that  the  optimization 
process  can  be  mechanized  through  an  economic  analysis  model,  the  judgment 
of  the  ATE  specialist  may  be  called  upon  in  the  final  selection,  if  only  to 
provide  inputs  to  the  model  in  the  form  of  risk  factors , degree  of  desirability 
of  excess  test  capability,  etc.  The  final  result  is  selection  of  the  optimum  ATE 
to  meet  a maintenance  concept , and  the  rationale  to  justify  the  selection  to  the 
eventual  user  of  the  equipment. 

4.4  FROM  SUPPORT  REQUIREMENTS  TO  SUPPORT  CONCEPT  DEFINITION 
4.4.1  General 

The  first  part  of  the  evaluation  procedure,  the  portion  between  entry  points  1 
and  2 on  Figure  4- 1 , is  expanded  and  described  in  this  section.  This  is  the 
only  time  in  the  acquisition  process  where  it  may  be  economically  feasible  to 
specify  built-in  or  on-line  test  methods.  At  this  time  plans  can  be  made  for 
integral  test  devices  or  for  an  interface  with  external  test  equipment.  Later  in 
the  acquisition  process,  the  addition  of  tost  equipment  provisions  can  become 
prohibitively  costly,  because  of  the  need  for  design  changes.  The  ideal  subject 
for  ATE  application  would  be  an  all  new  weapon,  radar  system,  or  radio.  New 
aircraft  and  ship  designs  usually  contain  a mixture  of  new  and  existing  sub- 
systems and  equipment,  which  complicates  the  ATE  selection  process,  pointing 
toward  a mix  of  manual  and  automatic  testing.  Figure  4-2  summarizes  the 
process  to  be  described  and  suggests  the  organizational  responsibility  for  each 
step.  Later  portions  of  this  section  will  elaborate,  where  necessary,  on  the 
steps  shown  in  Figure  4-2,  and  which  are  briefly  discussed  below. 
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ON  LEVEL  OF  DETAIL  AT  FIR5T  STEP. 


SYSCOWlETSO  AS  REQUIRED. 


SYSCOM 


SYSCOMAETSO  AS  REQUIRED. 


Figure  4-2.  Support  Concept  Definition 


(1)  Define  System/Equipment  to  be  Supported  - A basic  first  step,  and 

the  responsibility  of  the  SYSCOM,  is  to  define  the  equipment  or  system 
to  be  supported. 

(2)  Postulate  Alternative  Integrated  Logistic  Support  (ILS)  - The  SYSCOM 
is  seen  as  performing  this  task,  which  essentially  consists  of  identi- 
fying ILS  concepts  for  comparative  evaluation.  Although  this  step  is 
not  a part  of  the  ATE  selection  and  evaluation  process , the  two  are 
mutually  dependent. 
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(3)  Define  Test  Equipment  Alternatives  - Definition  of  test  equipment 
alternatives  may  require  that  SYSCOM  be  assisted  by  TETSO  or  other 
technical  groups  with  ATE  backgrounds,  unless  the  SYSCOM  has  the 
capability  in-house.  Possibly,  a number  of  test  equipment  alternatives 
may  be  suggested  for  each  alternative  ILS  concept.  The  detail  to  which 
ATE  alternatives  can  be  specified  will  depend  on  the  level  of  detail 
available  to  describe  the  equipment  to  be  supported  (first  step  in 
Figure  4-2). 

(4)  Select  Best  Test  Equipment  Alternative  - A cooperative  SYSCOM/ 
TETSO  task  is  envisioned  to  select  the  best  of  the  test  equipment 
alternatives  for  each  alternative  ILS  concept.  Selection  criteria  will 
include  ILS,  cost,  and  technical  factors. 

(5)  Select  ILS  Concept  - Although  not  strictly  a step  in  the  ATE  selection 
and  evaluation  process,  the  selection  by  SYSCOM  of  the  maintenance 
and  related  integrated  logistic  support  concepts  will  determine  the 
ATE  concept,  since  the  ILS  alternatives  will  each  have  included  an 
ATE  alternative  from  the  previous  step. 

(0)  Define  Support  Concept  - This  is  essentially  the  paper- work  task  of 
defining  the  previously  selected  ILS  concept  in  terms  which  can  be 
understood  by  those  concerned  with  the  ILS  interface.  The  related 
task  of  specifying  the  ATE  is  seen  as  requiring  an  ATE  background. 

4.4.2  Define  System  /Equipment  to  be  Supported 

The  validity  of  the  ATE  selection  and  evaluation  process  depends  largely  on  the 
level  of  detail  to  which  the  supported  equipment  can  be  described.  The 
supported  equipment  could  consist  of  an  aircraft  or  ship  subsystem  or  group 
of  subsystems.  It  could  comprise  an  entire  missile  weapon  system,  or  simply. 
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an  individual  equipment.  Figure  4-3  expands  the  single  step  shown  at  the 
beginning  of  Figure  4-2  into  a number  of  levels  of  detail.  The  levels  are 
arranged  in  order  of  increasing  detail  going  fro^r.  top  to  bottom,  and  their 
significance  is  discussed  below. 


Budget  - Cost.  Space,  Weight,  Power;  These  are  typical  budgetary  parameters 
which  apply  to  the  equipment  to  be  supported.  A bound  on  acquisition  cost  can 
be  eased  by  proving  a life  cycle  cost  reduction  which  exceeds  the  acquisition 
cost  increase  due  to  the  proposed  ATE  by  a sufficient  margin  to  make  a 


HIGH  PROBABILITY  THAT  INFORMATION  IN  FIRST 
FOUR  CATEGORIES  WILL  BE  AVAILABLE  DURING 
ILS  CONCEPT  DEFINITION  PHASE 


POSSIBLY  AVAILABLE  DURING  ILS  CONCEPT 
DEFINITION  PHASE 


OF  DOUBTFUL  AVAILABILITY  DURING  ILS  CONCEPT 
DEFINITION  PHASE,  EXCEPT  FOR  EXISTING 
EQUIPMENT 


AS  ABOVE 


Figure  4-3.  Define  System /Equipment  to  be  Supported 


4-10 


credible  case.  Space,  weight,  and  power  constraints  are  more  confining, 
particularly  in  airborne  applications.  However,  continuing  micro-circuit 
developments  promise  significant  future  size  reductions. 

Reliability,  Maintainability,  Availability  (RMA):  Early  in  the  development 
phase,  RMA  factors  will  exist  only  as  predictions,  if  at  all.  Reliability  and 
maintainability  factors  should  be  used  conservatively.  Mean-time-between- 
failure  (MTBF)  and  mean-time-to- repair  (MTTR)  are,  by  definition,  means. 
Actual  failure  and  repair  times  can  deviate  from  those  means.  Also,  because 
of  human  error,  skill  level  variations,  and  other  factors,  operational  experience 
is  apt  to  be  worse  than  the  predictions.  Operational  availability  will  be  based 
on  mission  scenarios,  and  is  likely  to  be  a firm  requirement. 

Performance  Requirements:  Performance  requirements  are  needed  to  syn- 
thesize end-to-end  performance  monitoring  systems,  whether  built-in  or 
external.  An  initial  idea  of  the  complexity  of  such  a system  can  be  estimated 
from  the  accuracy,  stimulus,  and  measurement  requirements  implicit  in  the 
performance  specification. 

I'unclional  Description  < )vonill  and  Segments:  This  level  of  definition  is  a 
further  expansion  of  performance  requirements.  Functional  descriptions  should 
get  down  to  segments  of  the  system  or  equipment  to  be  supported.  From  this 
information  it  is  possible  further  to  define  BITE  devices  and  to  start  defining 
off-line  ATE  alternatives  for  lower  levels  of  testing.  If  no  further  definition 
of  the  supported  equipment  can  be  made  available,  this  level  of  information  will 
enable  module  breakdown  estimates  to  be  made  by  an  engineer  who  is  familiar 
with  similar  existing  equipment. 
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Equipment /Module  Breakdown:  At  this  level , the  hardware  configuration  of  the 
supported  equipment  will  start  to  be  defined.  For  equipment  in  the  planning 
phase,  it  is  unlikely  that  this  level  of  definition  will  exist,  except  as  similari- 
ties to  existing  equipment.  During  the  Conceptual  or  Advanced  Development 
Phases,  this  information  can  be  synthesized,  but  changes  (20  - 50%)  can  be 
expected  during  the  Validation  or  Engineering  Development  Model  Phases. 
Generation  of  this  information  will  enable  medium  confidence  level  synthesis 
of  alternative  ATE  configurations  suitable  for  module  level  repair. 

I/O  Parameter  and  Operating  Tolerances:  The  input /output  parameters  and 
operating  tolerances  for  equipment  assemblies  and  modules  are  necessary  for 
synthesis  to  a high  confidence  level  of  ATE  to  the  module  repair  level.  This 
information  will  define  measurement  and  stimulus  requirements  and  accuracies. 
Equipment  would  have  to  be  well  along  in  design,  perhaps  75  to  100%  complete, 
before  reliable  module  operating  parameters  could  be  available.  Obviously,  it 
is  highly  improbable  that  these  details  would  be  known  for  new  developments, 
during  the  ILS  concept  definition  period , except  on  the  basis  of  close  similarities 
to  existing  equipment. 

Mechanical  Layouts  and  Schematics:  Mechanical  layouts  and  schematics  are 
essential  to  the  design  of  ATE  interface  hardware  internal  to  the  supported 
equipment,  of  external  adapters,  and  the  ATE  interface.  This  information  will 
only  be  available  on  existing  designs.  However,  for  planning  and  costing  of 
alternative  ATE  configuration  for  new  development,  it  should  be  possible  to 
estimate  these  items  based  on  past  experience.  The  estimating  errors  may  be 
high  but  should  not  be  significant  in  the  overall  cost  context. 

Availability  of  Information;  There  is  a high  probability  that  the  first  four  of  the 
seven  levels,  (that  is,  down  through  functional  descriptions)  of  supported 
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equipment  definition  previously  discussed  will  be  available  during  the  ILS 
concept  definition  phase  of  a new  equipment.  By  the  time  the  remaining  three 
levels  have  been  determined,  the  ILS  concept  will  already  have  been  defined, 
and  the  supported  equipment  will  be  heavily  into  the  Validation  Phase. 

4.4.3  Define  Test  Equipment  Alternatives 

General:  For  each  alternative  maintenance  concept  there  may  be  more  than  one 
test  equipment  alternative.  An  extensive  shipboard  installation  where  repair 
to  the  module  level  is  being  considered  could  include  an  ATE  to  fault  isolate 
to  the  module,  and  a separate  module  tester  to  locate  the  faulty  component  on 
the  module.  A single  ATE  system  could  also  handle  both  levels,  or  a separate 
module  tester  could  be  used  in  conjunction  with  BITE  to  locate  the  faulty 
module.  Possibly , the  faulty  module  could  be  located  by  manual  methods , 
aided  by  convenient  access  to  test  points. 

The  total  number  of  ATE  alternatives  need  not  increase  proportionally  to  the 
number  of  maintenance  concepts.  A considerable  degree  of  ATE  commonality 
can  exist  among  maintenance  alternatives.  Identical  module  testers  could  be 
proposed  for  use  on  shipboard  (Intermediate  Maintenance  Activity)  or  at  the 
depot. 

Figure  4-4  outlines  the  process  for  defining  test  equipment  alternatives.  The 
process  can  be  iterative,  as  shown,  with  feedback  which  can  alter  the  original 
maintenance  concept.  Although  this  portion  of  the  procedure  is  concerned  with 
ATE  selection  during  the  ILS/maintenance  concept  definition  phase,  the  same 
sequence  of  tasks,  but  in  much  greater  depth,  is  required  further  along  in  the 
acquisition  of  the  supported  equipment. 
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Figure  4-4.  Process  for  Defining  Test  Equipment  Alternatives 


Generic  ATE  Types:  The  first  step  consists  of  proposing  generic  ATE  typos 
compatible  with  the  particular  maintenance  alternative  under  study.  The  range 
of  ATE  types  will  depend  as  much  on  the  type  of  equipment  to  be  supported  as 
it  will  on  the  maintenance  alternative.  A wide  range  of  options  is  open  (and 
necessary)  for  systems  consisting  of  a number  of  equipments,  as  compared  to 
individual  equipments.  Typical  systems  which  may  be  encountered  are  ship's 
communication,  ship's  machinery,  shipboard  tracking  radar,  terrain  following 
radar,  and  air  defense  systems.  (The  dividing  line  between  an  equipment  and  a 
system  is  arbitrary  and  not  particularly  significant.  A system  can  be  considered 


to  be  an  operating  entity  consisting  of  more  than  one  equipment. ) The  possible 
generic  ATE  options  for  a system  can  be  made  up  from  one  or  a combination  of 
the  following: 

(1)  Built-in  test  equipment  (BITE) 

(2)  other  on-line  test  systems 

(3)  Off-line  test  systems 

(4)  Module  testers 

In  addition,  the  generic  ATE  options  above  can  each  be  further  broken  down  into 
one  or  a combination  of  the  following  optional  operational  modes: 

(1)  On-line  (noninterference  with  equipment  operation  during  test) 

(2)  Off-line  (opposite  of  on-line) 

(3)  Performance  monitoring  (end-to-end  test) 

(4)  Various  fault-isolation  levels  - segment,  assembly,  module,  component 

Identify  On-Line  Test  Requirements:  The  mission  will  have  to  be  analyzed  to 
determine  the  need  for  on-line  monitoring.  Prom  an  operational  and  mainten- 
ance viewpoint  on  lino  monitor  inr.  (or  IchUuk)  Ih  (lie  most  denindiV  mode  nf 
operation.  However,  rout  and  leelmmd  eon«idenil ions  could  require  Hint  on 
line  monitoring  be  limited  to  mission-critical  parameters.  On-line  testing 
can  be  specified  for  end-to-end,  segments,  assemblies,  and  even  modules, 
where  it  is  technically  possible. 

Determine  Level  of  Off-Line  Test  Desired:  Many  of  the  comments  made 
regarding  on-line  test  requirements  are  also  applicable  to  off-line  testing.  The 
major  difference  is  that  the  lowest  level  of  fault  isolation  - down  to  the 
component  - can  be  specified  with  a much  higher  degree  of  confidence  for 
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off-line  test  equipment  operating  in  an  off-line  mode  than  for  on-line  configura- 
tions. The  degree  of  fault  isolation  attainable  in  off-line  operating  modes  has 
so  far  tended  to  be  least  on  BITE,  increasing  with  centralized  external  test 
systems,  and  achieving  a maximum  on  depot  type  test  sets  or  module  testers. 
Exceptions  are  possible  in  individual  cases,  but  a thorough  cechnical  under- 
standing of  the  equipment  to  be  monitored  and  of  ATE  technology  is  necessary 
to  avoid  over-specifying  test  requirements. 

Analyze  Test,  Availability  Requirements,  Workload:  To  analyze  test  require- 
ments in  detail  sufficient  for  precise  specification  of  ATE  and  its  software 
requires  information  on  the  supported  equipment  which  will  not  be  available 
until  it  is  almost  completely  designed.  Therefore,  until  that  time,  this  step 
will  have  to  rely  very  heavily  on  the  judgment  of  an  ATE  specialist  who  is  also 
familiar  with  the  proposed  design  of  the  equipment  to  be  supported.  As  pre- 
viously mentioned,  new  ships  and  aircraft  will  use  significant  amounts  of 
hardware  already  in  inventory.  However,  BITE  and  centralized  testing  (except 
on  a gross  end-to-end  basis)  will  usually  be  ruled  out  for  existing  equipment 
because  of  the  high  cost  of  modification  for  either  test  approach.  Centralized 
testing,  which  requires  the  lesser  modification,  still  requires  an  often  costly 
and  space-consuming  integral  interface.  For  those  equipments  it  will  be 
possible  to  determine  detailed  test  requirements  for  maintenance  levels  where 
off-line  ATE  can  be  used.  Equipment  still  in  the  planning  stage  will  require 
the  ATE  specialist's  judgment  to  identify  stimulus,  measurement,  switching, 
and  computational  requirements,  and  these  obviously  can  enable  ATE  specifica- 
tion to  the  A level  (MIL-STD-490)  at  best,  because  of  the  preliminary  state  of 
the  equipment  design. 

The  desired  impact  of  ATE  on  MTTR  must  also  be  analyzed.  This  will  require 
acquisition  of  projected  reliability,  maintainability,  and  availability  data. 
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In  effect,  a figure  will  be  derived  which  states  the  turnaround  time  for  a UUT 
to  be  processed  through  an  off-line  ATE  installation.  (The  time  to  detect 
faults  with  BITE  or  other  on-line  ATE  is  usually  negligible. ) A total  workload 
can  then  be  determined  by  adding  individual  processing  times  for  each  UUT. 

The  result  will  be  an  indication  of  desired  test  rate,  which,  because  of  set-up 
and  other  irreducibles  could  require  a number  of  identical  test  systems  or  a 
mixture  of  different  types  (e.g. , VAST  plus  x module  testers). 

Match  Test  Requirements  Against  ATE  Inventory:  At  this  point  a set  of  test 
requirements  will  exist.  The  level  of  detail  will  depend  on  the  depth  of 
technical  information  available  on  the  supported  equipment,  the  time  available 
to  assemble  the  test  requirements , and  the  judgment  (and  prescience)  of  the  ATE 
specialist.  If  a data  bank  exists  to  describe  current  ATE  inventory,  the  test 
requirements  should  be  in  the  same  format  to  facilitate  comparison  of  require- 
ments with  available  capability.  This  comparison  can  be  done  by  the  specialist, 
or,  if  the  complexity  of  the  task  warrants  the  cost  of  automation,  it  can  be  done 
by  a computerized  data  bank  and  a comparison  algorithm.  The  results  of  the 
comparison  will  fall  into  the  following  categories: 

(1)  Locution  of  one  or  more  candidates 

(2)  Identification  of  one  or  more  near  matches 

(3)  Nothing  suitable  in  inventory 

When  Inventory  is  Inadequate:  When  a comparison  of  test  requirements  against 
ATE  inventory  discloses  nothing  entirely  suitable,  the  ATE  specialist  has 
several  choices.  He  can  locate  the  best  match  to  his  requirements  and  consider 
modifications  to  the  inventory  item  to  eliminate  the  shortcomings.  He  can 
accept  the  inventory  ATE  as  is  and  reduce  his  requirements.  He  can  specify 


an  all- new  design.  Even  when  a suitable  ATE  is  found  in  inventory,  a new  or 
modified  design  might  also  be  considered  because  of  cost,  size,  over-capabil- 
ity, etc. 


Change  Feedback;  At  this  point  a number  of  ATE  candidates  exist.  They  have 
yet  to  be  evaluated,  and  a final  selection  made.  However,  the  degree  to  which 
the  originally  proposed  generic  types  and  test  requirements  are  technically 
feasible  will  at  least  be  apparent.  Original  goals  can  then  be  raised  or  lowered 
to  minimize  risk  and  more  closely  to  match  technical  realities. 


4.4.4  Selection  of  Test  Equipment  From  Alternatives 


The  basic  evaluation  criteria  for  selecting  an  ATE  system  for  a particular 
maintenance  concept  are  cost  and  supported  system /equipment  availability. 
There  are  many  factors  to  be  considered  in  a trade-off.  These  are  tabulated 
in  Figure  4-5  and  are  amplified  in  following  sections.  Each  factor  must  be 


IMPACT  ON  UUT  ACQUISITION  COSTS 
IMPACT  ON  UUT  MAINTENANCE  AND  SPARES 
COSTS 

IMPACT  ON  LIFE  CYCLE  COST 
IMPACT  ON  AVAILABILITY 
TECHNICAL  FEASIBILITY/RISK 
SCHEDULE  FEASIBILITY/RISK 
ATE  ACQUISITION/REACQUISITION  COSTS 
HARDWARE 
SOFTWARE 
UUT  INTERFACE 
SPARES 

DOCUMENTATION 

ATE  RELIABILITY,  MAINTAINABILITY, 
AVAILABILITY 

ATE  MANNING  AND  SKILL  LEVELS 

SPACE 

WEIGHT 

POWER 

GROWTH 


Figure  4-5.  Selection  of  Test  Equipment  From  Alternatives 
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weighted  by  the  evaluator  to  suit  the  particular  application.  The  SYSCOM, 
which  has  the  ultimate  responsibility  for  cost,  system  availability  and  perfor- 
mance, should  be  heavily  involved  in  the  evaluation.  The  SYSCOM  can  best 
determine  how  much  it  can  pay  for  a given  increase  in  availability,  or  for  future 
growth  capability,  or  how  acceptable  a proposed  increase  in  acquisition  cost  may 
be  for  a projected  life  cycle  cost  saving.  Evaluation  of  technical  risks  and  the 
creation  of  work-around  paths  will  require  the  combined  technical  and  manage- 
ment ingenuity  of  an  ATE  specialist  and  the  SYSCOM. 

Impact  on  Supported  Equipment  Costs:  The  introduction  of  ATE  will  result  in 
individual  positive  and  negative  impacts  on  supported  equipment  costs.  Where 
the  net  impact  is  an  increase,  the  ATE  alternative  will  be  acceptable  only  if  a 
compensating  advantage  can  be  demonstrated.  In  the  case  of  AEGIS,  the 
incorporation  of  ORTS  as  an  on-line  test  system  undoubtedly  increased  the 
AEGIS  system  acquisition  cost,  but  without  it,  availability  would  have  been 
unacceptable.  A description  follows  of  supported  equipment  cost  categories 
which  can  be  influenced  by  ATE  alternatives. 

Acquisition  Cost:  ATE  will  drive  the  acquisition  cost  of  supported  equipment 
upward  when  ATE  must  be  specially  procured  or  built-in  to  the  supported 
equipment.  Where  an  ATE  will  already  exist  for  use  elsewhere,  but  which  can 
also  accommodate  the  supported  equipment  under  study,  acquisition  costs  can 
still  increase  if  hardware  must  be  added  either  internally  or  externally  to  the 
supported  equipment  to  provide  an  interface  with  the  ATE  system,  or  where 
additional  software  is  required.  Where  BITE  or  an  ATE  interface  is  being 
considered  internally  to  the  equipment  to  be  supported,  the  individual  cost  is 
multiplied  by  the  production  quantity.  Difficult  to  justify,  unless  spectacular 
results  are  indicated,  is  the  modification  of  existing  equipment,  where  the 
modifications  will  only  affect  part  of  the  population  of  that  equipment.  This  is 
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typical  of  the  situation  where  a centralized  test  system  is  being  contemplated 
for  production  communication  gear  on  a ship.  The  modification  design  will 
encounter  technical  problems  in  the  way  of  space,  weight,  power,  and  electro- 
magnetic compatibility.  Then,  further  costs  will  be  incurred  for  a requalifica- 
tion program  and  a new  documentation  and  spares  package  to  support  what 
might  end  up  as  a new  equipment  designation. 

ATE  can  also  drive  acquisition  cost  downward,  when  improvements  in  avail- 
ability accruing  from  use  of  ATE  can  reduce  the  need  for  redundancy,  or  reduce 
the  production  quantity  originally  planned  on  the  basis  of  a longer  MTTR  without 
ATE. 

Maintenance  and  Spares  Costs:  ATE  will  tend  to  reduce  maintenance  and 
spares  costs.  Maintenance  manpower  skill  levels  will  be  reduced  by  more 
rapid  automatic  detection  and  isolation  of  faults,  which  reduces  the  time  and 
simplifies  the  task  of  repair.  Spares  costs  can  be  reduced  by  isolating  faults 
to  a level  which  might  be  impracticable  to  achieve  manually,  thus  reducing  the 
complexity  of  the  spares  to  be  stocked.  Further  cost  savings  can  be  effected 
by  shortening  the  turnaround  time  through  the  use  of  ATE  at  the  depot  or  inter- 
m exhale  facility,  thus  reducing  spares  pipeline  times  and  quantity  required. 

Life  Cycle  Cost:  Life  cycle  cost  calculations  will  include  all  costs,  including 
related  ATE  costs,  to  acquire  and  support  the  equipment  for  its  estimated  life. 
ATE  will  tend  toward  reducing  life  cycle  cost  because  of  maintenance  savings. 
However,  this  must  be  balanced  against  ATE  costs.  The  value  of  improve- 
ments in  availability  are  difficult  to  quantize  unless  a clearly  identifiable  re- 
duction in  supported  equipment  production  quantity  can  be  identified  from  the 
use  of  ATE.  However,  the  SYSCOM  should  place  a value  on  availability 


improvements  when  considering  life  cycle  cost.  There  is  an  obvious  political 
problem  in  balancing  acquisition  cost  increases  against  life  cycle  cost  reduc- 
tions. The  case  for  such  an  increase  must  be  clear  and  credible.  The  agency 
responsible  for  the  acquisition  budget  may  be  reluctant  to  support  an  increase 
now  in  exchange  for  a saving  in  the  future,  when  the  future  saving  will  accrue 
to  another  agency. 

Direct  ATE  Costs:  If  an  ATE  is  located  in  inventory  which  meets  requirements, 
its  reacquisition  cost  will  have  to  be  estimated.  If  the  ATE  is  no  longer  in 
production,  the  hardware  acquisition  cost  can  far  exceed  the  original 
manufacturing  cost  because  of  start-up  and  inflation  costs.  A new  ATE  can  be 
estimated  on  the  basis  of  cost  information  for  existing  ATE , modified  by 
inflation  escalation  factors  and  allowances  for  new  technology.  Software  costs 
will  also  need  to  be  estimated.  Software  cost  can  vary  considerably  from 
machine  to  machine  for  a given  unit-under-test  (UUT).  The  existence  of 
software  preparation  aids,  such  as  compilers  and  standardized  languages,  are 
potential  means  for  reducing  the  cost  of  preparing  individual  programs.  The 
creation  of  software  aids  is  usually  very  costly,  as  compared  to  almost  any 
test  program.  A rule  of  thumb  sometimes  used  is  to  estimate  $100. 00  per 
test,  where  a test  is  defined  as  a stimulus-response  combination  or  a compar- 
ison-go/no go  decision.  Digital  test  programs  can  be  reduced  to  around  $10. 00 
per  test  using  automatic  test  program  generation  methods. 

The  UUT  interface  can  be  a significant  cost  item.  For  off-line  ATE,  the  inter- 
face consists  of  a connecting  cable  and  usually  an  adapter  box  or  fixture.  The 
adapter  box  will  include  dummy  loads  and  sometimes  considerable  electronic 
gadgetry  which  is  too  special  in  nature  to  warrant  integrating  into  the  ATE.  A 
minimum  interface,  then,  could  cost  under  $100.00.  The  maximum  can  run  as 
high  as  the  benefits  will  justify  - easily  into  the  thousands  of  dollars. 


Support  costs  in  the  way  of  spares,  documentation,  and  maintenance  manning 
requirements  for  ATE  should  also  be  considered. 

Technical  Factors:  The  ATE  RMA  figures  are  important  evaluation  factors. 

A complex  ATE  is  subject  to  the  same  statistical  reliability  hazards  as  a 
complex  supported  equipment.  The  difference  is  that  a self-test  function  is 
inherently  easy  to  build  into  ATE.  Reliability  and  maintainability  will  influence 
maintenance  manning  costs,  previously  mentioned.  They  will  also  determine 
availability  and,  therefore,  UUT  turnaround  or  flow  rate. 

The  technical  risks  must  also  be  evaluated.  An  over- ambitious  set  of  technical 
requirements  could  result  in  technical  problems  which  are  impossible  or 
impracticable  of  solution  because  of  time,  money,  and  state-of-the-art 
constraints.  Where  a serious  risk  is  recognized,  requirements  may  have  to 
be  reduced.  Where  the  risk  is  marginally  acceptable , an  escape  route  should 
be  plotted  which  will  allow  for  a change  of  plan  just  before  the  point  of  no 
return. 

Space,  weight,  and  power  requirements  for  ATE  can  vary  from  trivial  to 
extremely  important  evaluation  criteria.  The  weighting  factors  obviously  depend 
on  the  application.  In  order  of  increasing  gravity,  they  are  depot,  aircraft 
carrier,  destroyer,  submarine,  aircraft. 

Growth  capability,  or  flexibility,  defined  as  capability  in  excess  of  current  needs 
or  as  provisions  to  facilitate  future  expansion,  can  also  be  an  evaluation 
criterion.  However,  this  factor  is  largely  subjective  when  it  comes  to  placing 
a dollar  value  on  it. 
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Technical  pressures  can  generate  a pronounced  bias  toward  the  very  latest 
techniques  and  make  it  difficult  to  reprocure  an  existing  system.  This  bias 
must  be  counterbalanced  in  the  ATE  evaluation  process  by  consideration  of  the 
possible  costs  and  the  benefits  of  the  new  technology  to  the  prime  equipment. 


4.5  MODELING  AIDS 


4.5.1  Logistics  Models  as  Tools  in  Early  Planning  for  Support 

Integrated  logistic  support  is  a composite  of  all  the  support  considerations  nec- 
essary to  assure  the  effective  and  economical  support  of  a system  for  its  life 
cycle.*  It  is  an  integral  part  of  all  other  aspects  of  system  acquisition  and  op- 
eration. ILS  begins  with  program  initiation  (conceptual  and  validation  phases), 
and  continues  through  full-scale  development,  production,  and  deployment. 

The  choice  of  test  equipment  is  inherent  in  the  ILS  process  and  the  alternatives 
available  to  the  acquisition  manager  are  not  the  same  for  each  phase  in  the 
life  cycle.  During  conceptual  effort,  the  focus  is  on  system  feasibility  studies 
and  the  major  output  is  the  basis  for  a decision  as  to  whether  a system  acqui- 
sition program  should  be  pursued.  Support  activity  during  the  conceptual  phase 
is  concerned  with  defining  the  maintenance  environment,  the  interface  with  the 
logistic  systems,  and  such  goals  as  MTBM  and  MDT.  That  is.  the  support 
activity  is  dealing  with  requirements  which  will  be  the  basis  for  later  selection 
of  test  equipment  to  meet  the  requirements. 

In  the  advanced  development  phase,  the  requirements  for  BIT/BITE  are  devel- 
oped and  baseline  maintenance  concepts  are  specified.  The  role  of  separate 
test  equipment  emerges  at  intermediate  and  depot  levels  in  relation  to  BIT/BITE. 

During  conceptual  effort  and  advanced  development,  various  simulation  processes 
are  used  to  anticipate  the  effect  of  different  approaches.  Firm  descriptive  data, 
(such  as  exact  frequencies  of  operation,  equipment  failure  rates,  etc.)  is  not 

1 Integrated  Logistic  Support  - Implementation  Guide  for  DoD  Systems  and 
Equipments  - NAVMAT  P-4000,  March  1972. 
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available.  The  relationship  of  many  variables  (such  as  failure  rate  as  a function 
of  equipment  utilization)  is  better  known  than  the  numbers  themselves.  In  such 
a situation,  simulation  processes  are  useful  in  giving  an  answer  to  "what  if" 
questions.  For  example,  what  if  the  required  operational  availability  is  so 
high  that  MDT  cannot  exceed  ten  minutes?  Or  what  if  there  were  no  intermediate 
level  of  support,  and  faulty  items  were  returned  from  the  fleet  user  directly  to 
depot?  The  tools  which  simulate  the  relationship  of  the  variables  in  a process 
are  called  models.  Models  are  a part  of  the  ILS  tools  of  the  trade.  They  are 
used  to  indicate  what  is  significant  and  what  is  not  in  planning  the  long  term 
support  of  an  equipment. 

They  can  anticipate  the  effect  of  suddenly  increasing  the  usage  rate  of  an  equip- 
ment by  a factor  of  2:1  or  the  savings  in  support  costs  of  achieving  an  additional 
10  percent  MTBF  during  design. 

The  logistician  uses  models  to  help  define  the  elements  of  logistic  support  in 
time  to  impact  design,  plan  for  equipment  deployment,  and  control  O&M  costs. 
Models  can  also  be  useful  to  those  concerned  with  implementing  the  support 
concepts  developed  by  the  logistician,  i.e. , those  concerned  with  selection  of 
test  equipment. 

4.5.2  Decision  Points  Which  Can  Be  Supported  By  Modeling 

Figure  4-6  indicates  the  phases  in  an  equipment's  life  cycle  and  the  test  equip- 
ment related  activities  are  indicated  in  the  flow  of  tasks.  There  are  many 
other  activities  which  are  not  shown;  the  activity  flow  was  drawn  especially  to 
emphasize  test  equipment  related  elements. 
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Figure  4-6.  Equipment  Life  Cycle 


In  the  conceptual  and  advanced  development  or  validation  phase,  test  equipment 
decisions  are  of  the  type  ”what  generic  kind  of  test  equipment  should  be  anti- 
cipated in  support  of  these  operational  and  maintenance  requirements”?  It  is 
too  early  in  the  life  cycle,  for  most  prime  systems,  to  select  test  equipment  A 
or  to  evaluate  test  equipments  A,  B,  and  C.  Rather,  the  range  of  decisions 
would  include  a choice  of : 

• Whether  an  operational  availability  can  be  better  achieved  with  built-in 
test,  separate  test  equipment,  or  a combination  of  both 

• Whether  board  and  subassembly  fault  isolation/ repair  can  more  effec- 
tively be  accomplished  using  manual  general  purpose  test  equipment, 
general  purpose  ATE,  or  special  purpose  ATE  unique  to  the  prime 
mission  equipment 

• Whether  overall  savings  accrue  from  using  the  same  test  equipment 
at  intermediate  and  depot  levels  although  the  testing  requirements  are 
different. 

Before  moving  to  full-scale  development,  the  support  concept  should  be  defined 
(level  of  repair,  built-in  test  decisions,  etc.).  Models  will  be  very  much  in 
evidence  as  the  support  concept  is  firmed  and  early  test  equipment  decisions 
are  made.  At  this  stage  input  data  consists  of  such  operational  parameters  as: 

• Availability 

• Utilization  rate 

• Deployment 

• Reliability 


and  such  program  requirements  as 


• Unit  cost  targets 

• Service  life 

A second  area  where  models  are  likely  to  be  useful  in  supporting  test  equipment 
decisions  is  during  development  and  production.  During  some  time  in  the  de- 
velopment phase,  a specific  test  equipment  must  be  selected.  This  test  equip- 
ment may  be  a developmental  requirement  described  by  an  equipment  specifica- 
tion; it  may  be  itself  in  development. 

The  input  data  available  to  the  model  would  consist  of: 

• Specific  prime  equipment  test  requirements  (frequencies,  voltage 
levels,  accuracies) 

• ATE/UUT  interface  requirements  in  terms  of  number  and  types  of 
connectors,  test  access  points,  power  requirements 

• Workload  requirements  in  terms  of  the  numbers  and  types  of  UUTs  per 
unit  time  which  arrive  at  the  test  station 

• Personnel  subsystem  characteristics,  operator  training  assessments. 

In  general  the  input  data  could  be  expected  to  be  quantitative,  with  an  accuracy 
confidence  derived  from  design  data  and  limited  test/evaluation.  Sensitivity 
analysis  would  be  used  to  identify  parameters  critical  to  the  ATE  selection. 

A third  application  for  support  system  modeling  is  when  the  overall  support 
resources  must  be  estimated.  Support  resources  are  all  the  elements  of  test 
equipment,  spares,  operator/maintenance  personnel,  tech  data,  periodic  cali- 
bration, etc.  which  are  needed  to  maintain  the  prime  mission  equipment.  After 
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an  equipment  is  deployed  with  the  fleet,  along  with  supporting  test  equipment, 
there  are  frequently  changes  in  the  number  of  equipments,  in  where  and  how 
they  are  used,  and  as  significant  modifications  to  the  basic  equipment.  When 
such  changes  are  contemplated,  modeling  techniques  are  useful  in  anticipating 
the  impact  of  change.  For  example,  increasing  the  number  of  equipments 
aboard  ship  may  change  the  test  workload  such  that  an  additional  test  equipment 
is  needed.  Models  are  used  to  anticipate  the  total  resource  requirements  re- 
sulting from: 

• A modified  support  concept,  such  as  off-loading  board  repair  to  inter- 
mediate or  depot  level  (such  as  Project  Blackball) 

• The  opportunity  to  replace  obsolescent  test  equipment  with  more 
efficient  newer  design 

• Major  modification  to  a shipboard  electronic  equipment,  which  results 
in  new  test  requirements 

• COSALS  for  a new  ship  class,  where  combined  test  requirements  are 
the  basis  for  shipboard  test  equipment  selection 

• Fleet,  data  indicating  significant  increase  in  prime  equipment  failure 
rates  and  therefore  reduce* I maintenance  workload. 

The  models  discussed  in  this  section  are  mathematical  models.  That  is,  the 
relationships  of  independent  and  dependent  variables  in  the  maintenance  process 
are  expressed  in  mathematical  terms  and  converted  to  a program  which  runs 
on  a general  purpose  digital  computer. 

There  are  two  basic  approaches  to  expressing  the  relationships  within  the  model 
and  each  approach  has  distinct  advantages  and  disadvantages  for  one  or  another 
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application.  The  two  approaches  are  identified  as  analytical  and  simulation 
models.  The  analytical  model  provides  a single  answer  or  set  of  answers  for 
a given  set  of  inputs.  The  analytical  model  gives  an  answer  for  all  input  values 
held  fixed.  The  simulation  models  trace  the  behavior  of  the  process  over  a 
time  period  during  which  key  variables  are  allowed  randomly  to  assume  some 
value  within  its  range  of  possible  values. 


4.5.3  Generic  Model  Types  and  Their  Characteristics 


Analytical  Models;  It  is  generally  a characteristic  of  analytical  models  that 
they  provide  solutions  with  relatively  small  computation  effort.  The  relation- 
ships between  various  factors  may  be  complex  but  most  likely  or  best  estimate 
values  are  used  to  obtain  a single  point  solution. 

2 

The  formulations  within  the  model  are  of  a type  in  this  simple  example: 

AFD  = IUD/(UD  + IUD) 


AFD  - an  apportioning  factor  for  support  equipment  at  depot 

IUD  = item  utilization  of  s upport  equipment  at  depot 

UD  = utilization  of  depot  level  repair  support  equipment  per  month 
on  other  items. 


A Preliminary  Report  on  the  Final  NAVSHEPS  Level  of  Repair  (LOR)  Model  - 


Virginia  Research,  Inc.  , February  1973. 
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AFD  is  expressed  as  a fraction  of  time,  such  as  .3  or  .4  while  HJD  and  UD  are 
in  hours  and  hours  per  month  respectively. 

Most  reliability  models  are  analytical  models , based  on  a formulation  relating 
the  probability  of  occurrence  of  various  events.  The  function  of  failure  rate 
prediction  is  closely  related  to  maintenance  workload  estimation. 

Other  analytical  models  are  useful  in  examining  processes  which  involve  the 
flow  of  material  between  geographic  locations,  such  as  repairable  items  and 
spares  stocks  between  maintenance  levels. 

It  is  a characteristic  of  analytical  models  that  computer  running  time  is  t:hort  - 
on  the  order  of  seconds  or  a few  minutes.  This  means  that  the  user  can  expect 
to  get  responses  quickly  and  cheaply,  and  he  will  be  able  to  iterate  various 
solutions  as  the  most  significant  characteristics  emerge  in  the  trial  solutions. 

Typically,  the  technique  of  sensitivity  analysis  is  used  within  analytic  models 
as  a way  of  handling  input  data  uncertainty.  The  classical  parametric  study  is 
an  example  of  sensitivity  analysis,  wb^re  a range  of  solutions  are  presented  for 
variations  of  a key  input  factor.  The  factor  is  allowed  to  assume  values  from 
a lowest  likely  to  a highest  likely  value.  Examples  of  sensitivity  analysis  will 
be  shown  in  the  section  on  input  data  requirements. 

Every  model  falls  short  of  being  an  exact  replica  which  correctly  relates  all 
factors  in  the  modelled  process  under  all  possible  conditions.  As  such  the 
model  is  a compromise,  attempting  to  provide  an  adequate  representation 
without  becoming  so  complex  or  so  demanding  of  input  data  that  it  is  unwieldy 
to  use.  A support  sy  *om  model  which  neglects  ATE  spare  parts  costs  would 
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probably  not  adequately  represent  significant  cost  elements;  a model  that  re- 
quired an  individually  priced  spare  parts  list  would  require  more  effort  than  it 
is  worth  in  generating  input  data. 

Typical  input  data  would  include  that  shown  in  Table  4-1.  The  input  data  re- 
quirements are  more  a function  of  the  process  being  modelled  than  of  the  type 
of  model.  Additional  information  about  input  data  is  provided  in  the  Input  Data 
Requirements  section. 


Table  4-1.  Example  Problem  Data  Base 


• Deployment  Factors  - Number  of  systems  supported,  geographical 
location,  utilization  rate,  support  hierarchy  to  include  relation  to 
organizational  structure,  and  work  week. 

• Equipment  Factors  - Equipment  breakdown,  units,  modules,  parts; 
failure  rates;  physical  characteristics;  operating  times;  and  costs  per 
unit,  module  and  part. 

• Supply  Factors  - Stockage  policies,  supply  times,  production  lead 
times,  stockage  costs,  and  transportation  factors. 

• UUT  Modifications  - Modifications  of  fleet  UUTs  and  the  provision 
quantity  during  the  operational  phase  of  the  program. 

• Test  Equipment  Factors  - Test  equipment  characteristics,  costs,  and 
support  maintenance  requirements. 


Analytical  methods  are  particularly  applicable  to  reliability  and  maintainability 
models , to  resource  allocation  models , to  life  cycle  cost  and  logistic  models . 
and  to  optimization  problems  where  the  best  of  alternatives  is  to  be  chosen. 

Simulation  Models:  "Systems  characterized  by  large  data  banks  or  sizable 
solution  sets  can  be  handled  with  simulation  models.  Simulation  traces  the 
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system's  behavior,  frequently  over  time,  under  a specific  group  of  constraints, 
such  as  initial  conditions,  exogenous  and  design  variables,  target  conditions, 
and  internal  structural  properties.  Functional  relationships  exist  between  the 
solution  parameters  and  the  control  or  state  variables  in  the  model , and  in 
some  cases  the  solutions  are  not  obtained  as  point  estimates  but  rather  as  in- 
tervals that  contain  the  correct  answer. 

Although  simulation  is  frequently  implemented  for  complex  situations,  it  is  not 
necessarily  true  that  the  solution  implied  from  a given  set  of  input  data  is 
optimal.  Instead,  it  represents  an  approximation  to  the  best  answer,  and  the 
modeler  must  introduce  various  input  combinations  to  compare  their  implications 
for  the  desired  goals  in  the  system  analysis.  Yet,  even  with  the  selection  of 
many  different  input  data,  the  attainment  or  realization  of  an  optimal  solution 
cannot  be  assured  as  it  is  for  the  analytical  approach. 

Although  simulation  is  generally  more  adaptable  to  large-scale  computational 
problems  than  analysis,  it  also  gives  approximate  solutions  whose  optimality 
may,  or  may  not,  be  justified  on  theoretical  grounds.  Further,  simulation 
models  arc  generally  larger,  more  difficult  to  debug  and  validate,  and  more 
expensive  to  run  than  analytical  models.  They  can  be  used,  however,  to  analyze 
situations  that  are  just  too  complex  for  analytical  models  to  handle.  They  are 
thus  exceedingly  useful  for  analyzing  complicated  systems  in  uncertain 
environments."3 

Simulation  models  are  particularly  useful  when  it  is  desirable  to  examine  a 
complex  process  operating  over  a long  time  frame.  For  example,  it  may  be 


3 Using  Logistic  Models  in  System  Design  and  Early  Support  Planning  - Rand 
Report  R-550-PR,  February  1971. 
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desirable  to  consider  the  maintenance  workload  at  a tender  based  intermediate 
maintenance  facility  over  a year's  time,  during  which  the  number  of  ships  to  be 
supported,  their  electronic  equipment  complement  and  time  at  sea  would  vary 
on  a day-to-day  basis.  Simulation  type  models  would  provide  a picture  of  the 
workload  ebb  and  flow,  showing  times  of  overload  and  times  of  very  little 
workload.  The  results  would  be  synthetic,  i.e. , they  portray  a situation  which 
could  occur  but  would  most  likely  not  be  precisely  duplicated  in  real  life.  There 
are  situations,  such  as  the  work  shop  example,  where  the  modeler  wishes  to 
simulate  a dynamic  situation  in  which  independent  variables  are  allowed  to 
assume  any  value  within  a likely  range. 

The  disadvantage  is  that  simulation  models  can  be  costly  to  run  because  of  their 
complexity  and  the  user  still  lacks  insight  from  the  solution  as  to  which  of  his 
parameters  have  the  greatest  impact  on  the  system. 

4.5.4  How  Models  Are  Used 

Although  the  way  models  are  used  depends  on  the  problem  or  process  being 
modelled  and  the  characteristics  of  the  model  itself,  the  general  procedure  is 
the  same  for  each  application.  The  general  procedure  for  using  models  rel- 
ative to  support  equipment  selection  is  shown  in  Figure  4-7. 

In  the  figure,  larger  blocks  such  as  "Describe  Problem  to  be  Modelled"  are 
composed  of  more  specific  steps.  The  first  step  in  describing  the  problem  to 
be  modelled  is  to  define  the  supported  equipment.  As  indicated  on  the  figure, 
this  includes  such  data  as  the  prime  system  configuration,  e.g.  , it  is  a ship- 
board radar  composed  of  seven  major  assemblies,  36  subassemblies,  and 
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Figure  4-7.  Support  Equipment  Selection  Model 
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450  plug-in  modules  of  210  different  types.  Other  data  describing  the  prime 
system  is  needed  including: 

Equipment  breakdown 

Failure  rates 

Physical  characteristics 

Cost  per  assembly/subassembly/module 

Required  availability. 

Operational  characteristics  of  the  supported  equipment  must  be  defined.  Oper- 
ational characteristics  are  related  to  how  and  where  the  prime  systems  are 
used.  This  includes  how  many  equipments  are  aboard  ship,  and  the  number 
of  equipments  in  the  fleet  (if  the  test  equipment  application  concerns  IMA  or 
Depot).  Also  needed  are  operational  factors  which  determine  workload,  such 
as  equipment  utilization,  periodic  maintenance/calibration  requirements,  etc. 

After  the  problem  statement  and  descriptive  data  on  the  supported  system  is 
available,  the  analyst  proceeds  to  identify  the  alternatives  to  be  evaluated. 

If  the  prime  equipment  is  in  the  conceptual  or  advanced  development  phase,  the 
alternatives  are  likely  to  be  aimed  at  validating  the  support  concept  - to  support 
such  decisions  as  whether  to  repair  modules  aboard  ship  or  off-load  to  an  IMA, 
or  whether  to  incoiporate  BITE  in  order  to  meet  an  availability  requirement. 

If  the  above  examples  are  the  nature  of  the  decisions  to  be  made,  each  alter- 
native must  be  defined  as  though  it  were  the  chosen  approach.  If  an  alternative 
is  using  built-in  test  to  detect  failures  and  isolate  them  to  a replaceable  sub- 
assembly  level,  this  approach  must  be  synthesized  in  such  detail  that  a quan- 
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titative  solution  to  such  effectiveness  measures  as  "impact  on  availability" 
and  "support  cost"  can  be  generated.  Figure  4-7  indicates  the  level  of  data 
to  be  developed. 

It  is  not  necessary  that  all  input  data  be  developed  or  that  it  be  extremely 
accurate  before  the  first  model  runs  are  made.  Many  models,  for  example, 
will  assume  a most  likely  or  "default"  value  for  some  input  factor  which  is  not 
known.  Most  likely  values  are  based  on  experience  with  similar  model 
applications. 

Most  models  also  contain  format  or  lack  of  data  error  messages  which  are 
particularly  helpful  in  the  initial  runs.  In  examining  the  initial  results,  the 
analyst  looks  for  extraordinary  values  which  indicate  a problem  in  synthesizing 
alternatives  or  an  unusual  sensitivity  to  some  parameter. 

The  next  steps  are  an  iterative  process  wherein  the  analyst  examines  model  out- 
put, uses  sensitivity  outputs  to  narrow  the  contending  alternatives  to  those  which 
are  clearly  the  most  likely  choices.  If  the  problem  is  selecting  a generic 
support  equipment  approach  (BITE,  separate  automatic,  separate  manual,  etc.) 
the  iterative  process  includes  reshaping  the  alternatives  and  the  recommended 
approach  is  often  an  amalgam  of  two  or  three  of  the  original  alternatives. 

The  final  steps  focus  on  the  selected  approach,  its  description  in  detail,  and 
presentation  of  the  supporting  data.  Although  the  raw  supporting  data  has  been 
developed  in  previous  model  runs,  it  remains  to  be  presented  as  bar  charts, 
nomographs,  trade-off  and  sensitivity  curves  which  illustrate  the  validity  of  the 
selected  approach  in  terms  of  such  measures  as  maintenance  hours,  develop- 
ment costs,  O&M  costs,  availability  impact,  etc. 
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If  modelling  is  used  in  support  of  decisions  to  be  made  at  entry  point  three  in 
the  overall  Test  Equipment  evaluation  process,  e.g. , which  of  several  candidate 
ATE  systems  will  best  serve  the  fleet  requirements,  the  alternatives  are  test 
equipment  A,  B,  C,  etc.  Input  data  defines  each  equipment  in  terms  of: 


+ cost  - hardware  and  software  acquisition,  operation  and  maintenance, 
etc. 

+ related  manpower  - operation  and  maintenance,  calibration,  etc. 

+ test  equipment  support  requirements  - spares,  technical  data,  oper- 
ating environment,  special  test  equipment,  software  generation 
facilities,  etc. 

I 

I 

In  all  alternatives,  the  scenario  should  be  the  same.  That  is,  all  alternative 
test  equipment  should  be  compared  on  the  same  basis.  Other  scenarios  may  be 
used  for  more  comprehensive  comparison  as  long  as  all  candidates  are  asked 

I 

by  the  model  to  support  the  same  workload  in  the  same  fleet  environment. 

In  the  third  modelling  application,  the  support  concept  is  fixed,  the  support 
equipment,  has  been  chosen  and  the  problem  concerns  anticipation  of  support 
resources  due  to  some  change  in  the  present  situation.  Such  changes  would 
appear  in  the  initial  step  of  describing  the  problem  (e.g. , modification  to  the 
prime  system  which  changes  the  number  of  assemblies)  or  in  the  second  step 
of  describing  alternatives  (e.g. , shipboard  workload  has  increased  to  where 
an  additional  test  station  is  being  considered). 


I 

I 


The  steps  of  iteration  and  refining  alternatives  apply  here  as  in  the  other 
modelling  applications.  However  since  entry  point  three  generally  applies  later 
in  the  acquisition  cycle . there  should  be  less  reliance  on  default  values  and 
greater  confidence  in  input  data  relative  to  workload,  hardware  costs,  and 
man-hour  information. 
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4,5.5  Input  Data  Requirements 


The  Process  Being  Modelled:  Sometimes  called  the  scenario,  the  overall 
maintenance  process  must  be  defined  in  order  that  the  model  can  put  the 
candidate  support  equipments  in  context.  The  model  must  simulate  the  candidate 
test  equipment  in  operation  as  a key  part  of  the  maintenance  process.  There- 
fore, the  maintenance  process  itself  is  an  input  requirement.  The  process 
involves  material  - prime  equipment  to  be  maintained,  spare  parts  for 
replacing  failed  items,  facilities  or  work  shops  in  which  test  and  repairs  are 
to  be  accomplished.  The  process  involves  geography  - the  location  of  ship- 
board equipment,  IMA,  and  depot  facilities  and  logistics  pipelines.  The 
process  involves  time  - the  response  time  from  request  to  receipt  of  a replace- 
ment, the  operating  time  of  the  supported  equipment,  and  the  service  life  or 
life  cycle. 

Standardized  Input  Data:  Much  of  the  specific  input  data  needed  by  the  model  is 
not  unique  to  the  problem  application.  There  are  maintenance  related  factors 
which  are  common  to  Navy  logistic  support.  For  example,  costs  of  packing 
and  shipping  spare  components  need  not  be  estimated  uniquely  but  such  numbers 
are  available  in  cost  handbooks.  Other  input  data  in  this  category  are: 

• Pipeline  times,  cost 

• Personnel  training  time,  costs,  turnover 

• Administrative  costs  associated  with  entering  new  items  in  supply, 
maintaining  items  in  supply,  requisition  costs. 

Historic  Data:  Some  input  information  has  a single  value  - e.  g. . the  cost  of  a 
repairable  module  may  be  $1200.  Other  input  information  will  assume  a range 
of  values  - e.  g.  , man-hours  to  test  and  repair  a unit  under  test.  In  developing 
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the  latter  Information,  prior  experience  on  the  same  or  similar  equipments  is 
sought.  If  the  modelling  application  is  evaluation  of  candidate  test  equipment, 
it  should  be  possible  to  obtain  historical  data  on  test  times,  MTBF,  etc.  on  the 
test  equipment.  For  new  equipment,  design  MTBFs  as  modified  by  historic 
data  on  similar  equipment  can  be  used. 

Data  Uncertainty;  The  analyst  can  expect,  and  anticipate,  that  he  will  never 
have  firm  inputs  for  all  his  data  requirements.  An  important  part  of  the  pro- 
cedure for  using  models  is  to  identify  quickly  those  factors  which  are  most 
significant  to  his  solution.  This  is  the  purpose  of  sensitivity  analysis,  where 
the  model  allows  some  parameter  to  vary  through  a range  of  values  while 
keeping  other  factors  constant  - at  their  most  likely  value.  By  examining  the 
change  and  rate  of  change  of  the  output,  the  analyst  can  judge  the  sensitivity 
of  his  problem  solution  to  the  changing  parameter. 

An  example  of  sensitivity  analysis  is  shown  in  Figure  4-8.  In  this  example 
the  varying  parameter  is  operating  time  fraction  - the  percent  of  real  time  that 
the  supported  equipment  is  operating.  This  parameter  directly  influences 
maintenance  workload.  The  vertical  axis  indicates  increasing  costs  of  support, 
as  a percentage  of  acquisition  costs.  The  various  curves  (e.  g. , "Module 
discard")  are  alternatives  being  evaluated.  It  is  clear  that  some  alternatives 
are  much  more  sensitive  than  others  to  changes  In  operating  time  fraction. 

"Unit  discard"  goes  off  scale,  beyond  200  percent  per  year,  for  an  operating 
time  fraction  of  about  15  percent.  IMA  repair  as  a very  stable  alternative  with 
little  change  over  the  entire  range  of  operating  time  fraction.  However,  depot 
repair  is  the  lowest  cost  policy  up  to  about  45  percent  operating  time  fraction. 

In  this  example,  if  the  analyst  expected  that  the  operating  time  fraction  would 
be  in  the  10  to  30  percent  range,  then  Depot  repair  is  the  likely  choice. 
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COST  OF  SUPPORT 
VS 

OPERATING  TIME  FRACTION 


This  example  applies  only  to  the  problem  for  which  it  was  derived  and  the 
analyst  must  resist  the  temptation  to  draw  conclusions  such  as  "IMA  repair 
is  a very  stable  policy".  Each  problem  has  its  own  solution. 

Logistics  related  models  are  useful  on  a much  broader  scale  than  selection  and 
evaluation  of  ATE..  Some  models  are  configured  specifically  to  support  decisions 
concerning  the  following: 

(1)  Level  of  repair  - making  a choice  among  alternatives  as  to  what  will 
be  repaired  and  where.  Repair/throw-away  decisions  ai*e  an  example 
of  this  area. 

(2)  Spares  - anticipation  of  sparing  quantities  and  their  efficient  location. 

(3)  Personnel  - anticipating  operator  and  maintenance  personnel  require- 
ments as  a function  of  failure  rates,  usage  rates,  etc. 
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(4)  Support  equipment  - determining  the  number  and  optimum  location  of 
test  and  repair  equipment,  generally  in  relation  to  prime  system 
availability  and  overall  cost. 


(5)  Life  cycle  cost  - calculation  of  total  cost  over  the  life  cycle  with  cost 
element  visibility  to  indicate  the  impact  of  alternative  logistic  choices 
(e.  g. , location  of  maintenance  facilities  or  selection  of  test  equipment). 

((5)  Operations  effectiveness  - determining  the  effect  on  prime  system 
operations  for  most  likely  operating  environment  and  for  potential 
variations. 

Each  logistic  related  model  could  have  some  usefulness  in  test  equipment 
decisions,  and  no  single  model  is  best  for  every  Navy  application.  However, 
there  are  some  guidelines  which  will  be  helpful  in  selecting  a model.  "Guide- 
lines" are  general  advice,  usually  accurate,  but  always  subject  to  being  sub- 
verted by  an  exception  from  time  to  time. 

(1)  Analytical  models  are  easier  to  use  than  simulation  models.  By 
"easier"  we  mean  that  useful  answers  can  be  acquired  more  quickly 
and  with  significantly  less  computer  time. 

(2)  Logistic  models  that  have  been  used  on  Navy  problems  will  have  an 
advantage  for  Navy'  users  over  similar  models  that  have  not  had  Navy 
exposure.  There  are  still  differences  among  the  services  with  regard 
to  maintenance- related  names  and  procedures,  making  it  worthwhile  to 
favor  a model  that  describes  input  data  as  "test  equipment"  rather 
than  "AGE"  or  whose  readout  describes  "IMA  cost"  rather  than 
"direct  support  cost". 


(3)  Rarely  will  a model  be  immediately  available  to  fit  precisely  your 
problem;  you  can  expect  to  do  some  minor  adaptation.  In  order  to 
adapt  a good  model,  to  decide  what  adaptation  you  really  need,  to  most 
efficiently  structure  the  input  data,  to  make  most  effective  the  inevitable 
process  of  iterations,  to  do  all  these  things  the  user  should  be  able  to 
work  with  an  in-house  systems  analyst/programmer.  The  analyst  is 

a bridge  between  the  logistician  (who  works  with  levels  of  maintenance, 
operational  availability,  and  turn-around  times)  and  the  computer 
programmer  who  thinks  in  FORTRAN  or  SIMSCRIPT  and  is  more 
concerned  with  CPU  time  than  system  down-time. 

(4)  Models  should  not  make  decisions;  models  should  support  decisions 
made  by  maintenance  managers. 

(5)  Models  are  most  useful  when  they  anticipate  a situation  before  it  exists. 
Therefore,  the  user  must  be  prepared  to  live  with  uncertainties,  best 
guesses,  planning  data,  extrapolations  from  historical  information, 
and  rules  of  thumb.  The  model  user  should  not  become  awed  by  the 
implied  precision  of  five  place  numbers  on  a computer  printout.  He 
should  maintain  his  focus  on  the  relative  effectiveness  of  several 
alternatives,  not  on  any  particular  absolute  value.  Computers  do  not 
lie,  but  neither  can  computers  discriminate  between  fact  and  fiction. 

(6)  Table  4-2  is  taken  from  the  Rand  Report  R-550-PR.  It  indicates  the 
range  of  logistics  models  in  use  and  gives  some  pertinent  character- 
istics of  each.  Where  model  type  is  indicated  "accounting"  is 
equivalent  to  analytical.  Since  the  Rand  report  was  prepared,  NADC 
has  adapted  CO-AMP  for  a number  of  Navy  Applications , as  shown 
in  Figure  4-9.  Figure  4-9  was  taken  from  the  forthcoming  NADC 
Report  73240-50  "N  \DC  Life  Cycle  Costing  Methodology  and  Applica- 
tions (1969  - 1973)". 
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4.6  A BUILT-IN  TEST  EVALUATION  MODEL  (BITEM) 


4.6.1  Background 

When  designing  BITE  for  a system  as  extensive  as  a tracking  radar  or  missile 
control  complex,  achieving  fault  isolation  uniformly  to  a single  module  or 
within  a fixed  number  of  modules  is  difficult.  Circuit  partitioning  for  such  a 
system  must  trade  standardization  against  ease  of  fault  isolation,  and  while 
logistic  considerations  govern  on  either  side  of  the  trade,  the  benefits  of 
maximum  standardization  can  justify  accepting  less  than  ideal  BITE 
performance.  This  occurred  on  AEGIS,  where  a supplementary  off-line  module 
tester  will  be  used  to  locate  the  faulty  modules  in  a group  identified  by  ORTS 
(ref.  Paragraph  3. 1.4)  as  containing  one  or  more  such  modules.  It  is  desir- 
able to  be  able  to  measure  the  performance  of  a proposed  BITE  configuration, 
especially  when  compromise  alternatives  require  evaluation.  The  scheme  to 
be  described  herein  was  devised  for  that  purpose.  It  is  the  outline  of  a model 
for  measuring  the  results  of  BITE  with  regard  to  how  closely  it  approaches  the 
ideal  of  fault  isolation  to  every  single  module  in  an  equipment,  and  comparing 
that  performance  to  its  cost.  Although  intended  for  evaluating  BITE,  a similar 
technique  could  be  applied  to  other  forms  of  test  equipment. 

4.6.2  Approach 

The  BITEM  approach  consists  of  developing  a number  which  represents  the 
summation  of  weighted  benefits  to  accrue  from  a proposed  configuration,  and 
then  evaluating  the  benefits  against  the  costs  to  achieve  them.  It  is  suggested 
that  life  cycle  costs  should  be  used,  although  limitation  to  acquisition  costs  is 
not  excluded.  The  expression  for  benefits  consists  of  the  algebraic  summation 
of  weighted  benefit  evaluation  factors.  Some  benefits  are  difficult  to  evaluate 
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numerically  with  precision,  and  a 0-4  scale  could  be  adopted  for  those  cases. 
Also,  there  are  negative  "benefits"  such  as  complexity,  size  and  power 
increases  due  to  BITE  which  are  subtracted  from  positive  benefits.  Of  course, 
the  negatives  would  go  positive  in  the  unlikely  event  that  BITE  should  result  in 
reducing  any  of  these  factors.  Net  negative  benefits  would  result  in  reject  jg 
the  proposed  configuration.  All  BITE  benefit  evaluators  will  consist  of  a 
number  which  represents  the  extent  of  the  benefit,  multiplied  by  a weighting 
coefficient  which  may  vary  from  application  to  application,  depending  upon  the 
relative  significance  of  each  benefit  to  the  particular  application. 

Costs  require  no  weighting  and  are  naturally  evaluated.  Costs  incurred  are 
positive;  cost  savings  are  negative.  Table  4-3  lists  benefit  and  cost  factors. 
Factors  are  self-explanatory,  but  will  be  further  enlarged  upon  later  in  this 
section.  Table  4-4  reduces  the  English  language  terms  to  a mnemonic  format. 


Table  4-3.  Built-In  Test  Evaluation  Model  (B1TEM)  Factors 

Benefit  = A (Performance)  + B(Availability  Increase)  + C (Flexibility) 

+ D(Growth  Capability)  + E(Man-Machine  Interface)  - F 
(Prime  Equipment  Complexity  /Size  /Power  Increase)  + G 
(Operational  Safety  Improvement)  + H(Mission  Assurance) 

Cost  (Life  Cycle)  = (Acquisition  Cost  Increase)  + (Support  Cost  Increasei  - 

(Saving  in  External  ATE  Hardware  Acquisition  and  Support) 
- (Saving  in  External  ATE  Software)  - (Saving  in  Prime 
Equipment  Maintenance  and  Spares) 

Evaluation  factors  are  in  parentheses;  those  for  benefits  should  be  scored. 


Benefit  weighting  factors  are  coefficients  A -II. 
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Table  4-4.  Built-In  Test  Evaluation  Model  (BITEM)  Mnemonic  Terms 


Benefit 

= A'PFRF)  + B(Aq) 

+ C(FLEX)  + D(GROWTH)  + E(ITF) 

- FfCOMFTEX!  + 

G(SAF)  + H(MSN) 

Cost 

= AACQ  + ASPT  - 

AXTE  - AXSW  - ALOG 

Figure  of  Merit 

- Benefit/Cost 

4.6  3 Benefits 


Table  4-5  tabulates  and  defines  the  benefit  factors.  An  attempt  at  numerically 
evaluating  test  performance  will  be  discussed  later.  The  availability  factor 
naturally  lends  itself  to  numerical  treatment,  as  do  complexity,  size  and 
power  increases.  Flexibility,  growth  capability,  man-machine  interface, 
operational  safety  improvement,  and  mission  assurance  are  not  so  easily 
evaluated,  and  discrete  steps  are  suggested  for  that  purpose.  A scale  of  0-4 
could  be  used  to  rate  the  degrees  to  which  these  benefits  are  in  evidence.  The 
numbers  0-4  would  be  equivalent  respectively  to  none,  poor,  fair,  good, 
excellent. 


Table  4-5,  BITEM  Benefit  Evaluation  Factors 


Performance 

Score  for  fault  isolation  level,  number  of  modules,  on-line,  off-line. 
Availability  Increase 

Desired  prime  availability  will  be  specified.  Scoring  will  be  negative  for 
availability  below  specification;  positive  above.  Weighting  for  availability 
above  specified  will  reflect  desirability  of  improvement. 

Flexibility,  Growth  Capability 

Flexibility  for  other  applications,  growth  scored  on  ease  of  expansion. 
Man-Machine  Interface 

Score  for  operational  convenience,  quality  of  displays. 
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Table  4-5,  BITEM  Benefit  Evaluation  Factors  (Cont. ) 


Prime  Equipment/Size /Power  Increase  (May  be  Three  Separate  Factors) 
Scoring  directly  proportional  to  increases.  Sign  changes  if  decrease 
achieved.  Weighting  significant  for  aircraft,  submarine  applications. 

Operational  Safety  Improvement 

Where  safety  not  a consideration,  weighting  can  be  zero.  For  flight 
applications  it  could  be  dominant  factor. 

Mission  Assurance 

Similar  to  operational  safety. 


4.6.4  Costs 


Table  4-6  tabulates  and  expands  on  cost  factors.  The  addition  of  BITE  will 
increase  acquisition  cost  as  it  increases  overall  equipment  complexity. 

Support  costs  will  also  rise  proportionally  as  an  incremental  increase  in 
spares,  maintenance,  and  documentation.  BITE  could  eliminate  or  reduc.  the 
cost  of  external  ATE  support  in  hardware,  software,  and  operational  and 
maintenance  areas.  Depending  on  the  level  of  fault  isolation  achieved, 
maintenance  manpower  and  spares  costs  could  be  reduced  on  the  prime 
equipment. 


Table  4-  6.  BITEM  Cost  Factors 


Acquisition,  Support  Increases 

Estimate  as  proportion  of  prime  in  absence  of  firm  data. 

Saving  in  External  ATE  Hardware 

Estimate  life  cycle  costs  of  external  ATE  hardware  eliminated  by  BITE. 
Saving  in  External  ATE  Software 

Estimate  saving  in  software  that  would  have  accompanied  above  ATE 
hardware.  If  ATE  hardware  available  at  no  extra  cost  (e.  g. , vast  on 
carrier!,  estimate  software  saved  by  not  having  to  use  vast. 
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Table  4-6.  BITEM  Cost  Factors  (Cont.) 


Saving  in  Prime  Equipment  Maintenance  and  Spares 

Estimate  saving  in  maintenance  manpower.  If  increase,  rather  than 
saving,  change  sign.  Estimate  saving  (or  increase!  in  support  spares. 


4.G.5  Performance  Benefits 


A major  benefit  is  that  of  BITE  performance  in  the  areas  of  monitoring  and 
fault  isolation.  An  attempt  has  been  made  in  BITEM  to  evaluate  this  factor. 

BITE  performance,  particularly  for  systems,  Is  often  a series  of  compromises 
with  the  ideal,  and  a method  has  evolved  in  BITEM  to  evaluate  those 
compromises.  An  abstract  example  is  shown,  consisting  of  six  modules  (or 
components,  or  assemblies),  which  represents  the  desired  level  of  fault  isolation 
by  BITE.  Although  it  is  preferred  that  BITE  will  isolate  faults  to  any  one  of  these 
six  modules,  it  is  recognized  that  technical  limitations  or  prime  equipment 
constraints  may  make  this  unfeasible,  and  that  faults  may  be  only  isolated  to 
groups  of  modules  in  some  cases.  Figure  4-10  shows  the  interrelationship  of 
the  six  modules.  Dotted  lines  enclose  the  fault  isolatable  groupings.  It  is 
assumed  that  only  one  grouping  at  a time  will  fail  or  be  fault  isolatable.  This 
is  not  an  immutable  law,  but  it  does  represent  the  usual  practice  with  BITE, 
which  makes  maximal  use  of  the  still  operational  portion  of  the  system  to  test 
the  remainder. 

Table  4-7  lists  the  scoring  rules.  The  modules  are  rated  in  complexity  on  a 
scale  of  1-4  and  are  scored  accordingly.  To  avoid  a lengthy  study  on  the 
distribution  of  components  and  individual  component  reliabilities,  this  could  be 
accomplished  lay  simply  counting  parts.  (Common  sense  would,  of  course, 
recognize  and  suggest  ways  of  handling  an  extreme  case  such  as  a module  full 
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of  resistors. ) Locating  a fault  to  any  individual  module  would  then  provide 
a score  of  1-4. 


Table  4-7.  Sample  Scoring  Procedure 


Performance 

Rate  modules  by  complexity  from  1-4. 

Score  1-4  for  each  individual  fault  isolatable  module. 

Where  modules  are  fault  isolated  as  groups,  totalize  complexity  numbers 
in  group,  and  divide  by  number  of  modules  in  group. 

Where  fault  isolatable  groups  overlap,  score  common  modules  only  in  the 
smallest  of  the  groups  in  which  they  appear.  Use  the  actual  module  count 
as  a divisor  for  the  complexity  total  of  the  remaining  modules. 

Use  multiplying  factor  of  1 for  off-line,  3 for  on-line,  or  weight  to  suit. 


Fault  isolatable  groups  would  be  scored  by  totaling  the  scores  of  all  the 
modules  in  that  group  and  dividing  by  the  number  of  modules  in  the  group. 
Where  groups  overlap,  common  modules  are  scored  only  in  the  smallest  group 
in  which  they  appear.  Hoover  the  scoring  divisor  for  any  group  will  always 
consist  of  a count  of  all  modules  in  that  group  regardless  of  where  else 
individual  modules  may  he  scored. 

Because  on-line  fault  isolation  (non-interfering  with  normal  operation)  is  more 
desirable  than  off-line  (interfering  with  or  interrupting  normal  operation),  a 
weighting  coefficient  of  3 is  assigned  to  on-line  fault  isolation.  This  is  an 
arbitrary  figure  which  can  be  vaiied  to  suit  the  particular  application. 

The  example  of  Figure  4-10  assumes  all  off-line  testing.  Group  A illustrates 
the  scoring  method.  Its  module  score  is  4 reduced  by  3 because  of  two  common 
modules  which  are  scored  in  groups  B and  C,  respectively.  The  division  of  3 
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COMPLEXITY  PERFORMANCE 

SCORE  DIVISION  SCORE 

GROUP  A 
B 
C 
D 
E 
F 

12,33 

MAX.  POSSIBLE  SCORE  - 15 


(4-3)  - 1 

3 

.33 

3 

2 

1.50 

(4-3)  = 1 

2 

.50 

3 

1 

3.00 

3 

1 

3.00 

4 

1 

4.00 

Figure  4-10.  Example  of  Performance  Scoring 


for  the  number  of  modules  in  Group  A results  in  a group  score  of  1/3,  or  0.  33. 
The  maximum  total  score  possible  would  have  been  15,  the  sum  of  all  individual 
module  scores.  However,  group  reductions  resulted  in  a score  of  12.33. 


This  model  is  not  concerned  with  the  fault  isolation  logic  or  method,  or  with 
the  mix  of  hardware  and  software  except  as  it  affects  cost.  It  was  intended  to 
provide  a much-needed  means  for  numerically  scoring  only  the  results 
achieved  by  BITE. 


There  are  possible  ambiguities  in  the  scoring,  and  the  model  can  be  manipulated 
to  exploit  them.  For  example,  although  it  might  be  possible  to  do  without 
Group  C in  a real-life  case,  and  with  a possible  software  saving  at  that,  its 
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retention  could  be  motivated  by  the  increase  in  score  from  11.83  to  12.33, 
although  the  cost  should  go  up  to  reduce  or  eliminate  the  advantage.  Clearly, 
this  model,  in  common  with  many  others,  does  not  entirely  eliminate  the  need 
for  human  judgment. 


SECTION  5 


ILLUSTRATIVE  EXAMPLES 
5.1  ACQUISITION  PHASES 

In  this  section  will  be  presented  examples  of  how  the  procedures  previously 
described  can  be  applied  during  each  acquisition  phase.  Case  histories  are 
used  to  demonstrate  the  importance  of  timing  and  the  interdependency  of  the  ILS 
and  the  ATE  definition  processes. 

No  ATEs  were  available  as  examples  which  were  acquired  under  the  new  acquisi- 
tion phase  terminology  of  NAVMAT  INST  4000.  20A.  Instead,  the  terms  Contract 
Definition,  Design  and  Development,  Engineering  Development  Model,  Production, 
and  Inventory  will  appear,  except  in  subsection  headings  where  the  new  terminology’ 
is  used.  Equivalences  follow. 

Old  NAVMAT  Inst  4000.  20A 

Concept  Formulation  Conceptual 

Contract  Definition  (CD)  Validation 

Design  and  Development  (D+D)  Full  Scale  Development 

Engineering  Development  Model  (EDM)  Full  Scale  Development 

Production  Production 

Inventory  Deployment ''Operational 

No  example  is  given  for  the  Concept  Formulation/Conceptual  Phases.  On  AEGIS 
it  was  observed  that  as  far  as  test  systems  were  concerned,  the  Navy  preferred 
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to  state  prime  system  requirements  and  allow  each  competing  contractor  a free 
hand  in  proposing  the  means  for  meeting  specified  availability  and  maintenance 
policies.  Then  the  Navy  evaluated  the  proposals  and  thus  selected  the  ATE 
approach.  For  the  Conceptual  Phase  of  large  systems,  such  as  AEGIS,  this 
seems  more  desirable  than  the  Navy's  specifying  the  ATE  system,  because  it 
encourages  creativity  in  contractors  and  gives  them  the  total  responsibility  for 
system  support.  For  smaller  individual  equipments,  such  as  radios,  which 
may  be  applicable  to  a variety  of  system  configurations  - aircraft,  surface 
ships,  submarines,  land  bases  - and  which  may  be  supported  by  different 
logistic  policies  in  each  environment,  it  is  recommended  that  the  Navy  specify 
the  generic  ATE.  The  Navy  is  in  a position  to  be  more  aware  than  the  contractor 
of  the  possible  range  of  future  applications  and  logistic  support  policies.  For 
radios  and  similar  equipments  BITE  will  Increasingly  be  specified  as  the  generic 
ATE,  and  the  contractor's  role  will  consist  of  striving  to  maximize  the  benefit- 
cost  ratio  of  the  BITE  design. 

5.1.1  Validation  Phase 

Introduction:  The  AEGIS,  Mark  7 weapon  system  contains  an  on-line  test  system, 
the  MK  545  Operational  Readiness  Test  System  (ORTS),  and  its  selection  followed 
the  procedure  described  in  this  report.  This  example  illustrates  the  ease  with 
which  ATE  selection  decisions  can  be  made  when  prime  equipment,  test  equip- 
ment, and  logistic  specialists  work  together,  and  when  trade-off  decisions  are 
simplified  by  a few  overriding  evaluation  factors. 

Three  contractors  were  selected  by  the  Navy’  for  the  Contract  Definition  Phase 
(now  Validation  Phase).  The  single  contractor  selected  to  design  and  build  the 
Engineering  Development  Module,  had  chosen  as  the  generic  ATE  type  an  integral 
on-line  system,  in  response  to  availability  and  maintainability  factors  which 
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were  specified  by  the  Navy,  based  on  operational  mission  requirements.  The 
ATE  selection  procedure  was  therefore  performed  by  the  contractor  in  this 
case,  with  Navy  approval  clearly  implied  by  review  and  acceptance  of  the 
proposal.  After  contract  award,  the  Navy  program  office  continued  to  contri- 
bute to  the  design  decision  process  by  means  of  design  reviews. 

Description  of  AEGIS:  The  AEGIS  weapon  system  includes  AN/SPY-1  multi- 
function radars,  illumination  radars,  and  Mark  26  missile  launchers.  The 
AN/SPY-1  radar  uses  an  electronically  scanned  phased  array  antenna  with 
individual  element  drive  amplifiers. 

The  illumination  radars  use  mechanically  steerable  antennas.  The  Mark  26 
launcher  is  digitally  controlled  for  automatic  selection,  load,  and  reload  of  the 
missile  types  (anti-air  or  anti-submarine)  required.  Illumination  radars  and 
missile  launchers  had  already  been  developed  when  the  AEGIS  program  was 
initiated.  The  AN/SPY-1  development  was  initiated  with  the  AEGIS  go  ahead. 
Therefore,  the  test  system  had  to  accommodate  a mix  of  test  philosophies, 
since  redesign  for  a uniformly  centralized  system  was  economically  unfeasible. 

The  ATE  Selection  Procedure;  The  first  block  in  Figure  4-1  is  establishment 
of  support  requirements.  This  was  done  by  the  Navy.  During  the  competitive 
CD  Phase,  RCA,  the  winner  of  the  EDM  phase,  synthesized  alternative  support 
concepts  and  evaluated  them  against  availability  and  maintainability  parameters 
developed  by  the  Navy.  Test  equipment  alternatives  at  the  generic  level  were 
also  synthesized.  The  extent  of  the  installation  - solid-state  equipment  compactly 
packaged  in  dozens  of  racks,  cabinets,  or  consoles  located  on  several  decks  - 
eliminated  manual  testing  from  serious  consideration  because  of  the  near 
impossibility  of  providing  the  high  degree  of  test  access  and  skilled  manpower 
which  would  have  been  called  for.  Off-line  ATE,  except  for  modules,  was  out 
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of  the  question  because  of  the  set-up  time  and  because  economies  were  indicated 
in  the  ability  of  an  integral  on-line  system  to  use  self-generated  stimulus  and 
other  portions  of  the  AEGIS  system  for  self-test  purposes.  It  very  quickly 
became  apparent  that  availability  and  maintainability  goals  required  an 
immediacy  in  status  reporting  and  fault  diagnosis  that  could  only  come  from 
an  automatic  on-line  test  system. 

Also,  the  system  was  designed  for  ''graceful"  degradation.  Individual  antenna 
element  drivers  could  fail  with  some  degradation  of  the  antenna  pattern,  but 
without  seriously  jeopardizing  overall  operation.  It  was  essential  for  the 
operator  to  be  aware  of  that  condition,  so  that  the  mission  could  be  modified 
if  necessary,  or  corrective  action  undertaken.  Data  on  existing  systems  of 
comparable  complexity  reinforced  this  decision.  It  was  clearly  impracticable 
for  the  on-line  system  to  fault  isolate  to  better  than  a module  or  group  of 
modules,  so  that  an  automatic  module  tester  w-as  indicated  to  hold  on-board 
spares  and  maintenance  manpower  to  reasonable  levels.  So  far,  it  is  seen  that 
the  decision  process  was  straightforward,  because  there  never  were  any  serious 
competitors  to  the  alternative  of  automatic  on-line  test  with  an  off-line  module 
tester  for  back-up. 

5.1.2  Full  Scale  Development 

Introduction:  Following  evaluation  of  Contract  Definition  Phase  results,  the 
contract  for  the  Engineering  Development  Model  was  awarded  to  RCA.  The 
ATE  selection  procedure  now'  stood  at  point  2 on  Figure  4-1.  The  support 
concept  and  generic  ATE  types  were  defined. 

Detailed  Test  Requirements:  Enough  preliminary  design  work  was  done  during 
CD  to  define  test  requirements  in  a general  way.  An  estimate  was  made  at 
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that  time  of  the  number  of  test  points  required  in  order  to  define  the  configura- 
tion and  cost  of  the  proposed  ATE.  During  design,  test  requirements  were 
developed  in  detail  to  enable  the  test  equipment  hardware  and  software  to  be 
better  defined.  To  accomplish  this,  it  was  necessary  to  make  test  logic  design 
a formal  part  of  the  equipment  design  process.  Otherwise,  the  normal  tech  :J  cal 
and  schedule  pressures  would  have  relegated  the  test  requirements  to  a low 
priority  and  to  possible  neglect.  The  equipment  designers  were,  by  program 
procedure,  required  to  locate  test  points  and  design  the  test  logic  which  showed 
how  the  test  points  were  going  to  be  used  and  what  the  logic  was  going  to 
accomplish.  Signal  conditioners  required  to  meet  the  ATE  interface  were  also 
identified.  This  information  was  documented  to  a predetermined  uniform  format 
and  reviewed  for  conformance  to  status  monitoring  and  fault  isolation  goals.  The 
review  groups  consisted  of  a mixed  panel  of  prime  equipment  design  personnel 
and  ATE  systems  engineers.  For  future  programs  the  review  panel  could  as 
easily  be  composed  of,  or  include  TETSO  or  other  Navy  representation,  with 
steps  taken  to  ensure  rapid  turn  around  where  the  need  is  identified  for  changes. 
The  same  discipline  was  exerted  on  the  subcontractor  who  was  designing  the 
AN/SPY- 1 radar  transmitters. 


Selecting  the  ATE  Configuration:  The  steps  that  begin  with  entry  point  no.  3 
in  Figure  4-1  are  described  here.  The  comparison  of  ATE  capability  to  test 
requirements  is  performed  by  the  prime  equipment  designer,  since  the  stimulus 
requirements  will  be  supplied  by  his  own  equipment,  and  the  measurement 
interface  will  already  have  been  defined  for  him  as  part  of  the  generic  ATE 
description.  The  reference  to  a data  bank  for  on-line  or  built-in  test  consisted 
of  examining  existing  data  multiplexing  and  transmission  methods  and  hardware, 
and  that  task  was  performed  by  the  ATE  systems  engineers.  The  candidates 
for  that  purpose  were  evaluated  by  the  ATE  systems  engineers  on  the  basis  of 
technical,  maintainability,  risk,  and  cost  considerations. 
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For  the  AN/SPY- 1 radar  the  selection  process  was  straightforward  because 
the  radar  and  the  ATE  were  designed  together.  The  illumination  radars  and 
missile  launchers  were  already  in  existence  with  their  own  integral  test  pro- 
visions, and  that  test  data  had  to  be  integrated  into  the  overall  ATE  system, 
which  eventually  was  designated  the  ME  545  Operational  Readiness  Test  System 
fORTSt.  The  alternatives  presented  were  to  modify  the  illuminators  and 
launcher*  into  a uniformly  centralized  test  system  or  to  usi  he  status  data 
internally  generated  by  them  in  die  best  way  possible  with  no  modification.  No 
formal  trade-off  was  required  to  eliminate  the  first  alternative,  at  least  for  the 
Engineering  Developmental  Mode.  The  cost  of  the  extensive  changes  that  wool  i 
have  been  required  did  not  justify  the  technical  advantages  that  would  have  bee  . 
achieved.  Status  panels  were  integrated  into  the  CRTS  Tern  and  Monitor 
Console,  which  contained  indicators  driven  by  the  illumination  radars'  internal 
status  monitoring  devices.  The  launchers  contained  integral  status  panels 
which,  it  was  decided,  would  continue  to  be  used  in  place,  with  possible  addition 
of  ORTS  data  collection  provisions  only  in  subsequent  models.  The  data 
processing  subsystem  was  based  on  standard  Navy  AN/UYK-7  computers  which 
depend  largely  on  self-test  software  for  fault  indication.  Control  of  self-test 
software  is  centered  at  the  ORTS  Test  and  Monitor  Console  to  permit  rapid 
reconfiguration  and  load  transfer  in  the  event  a computer  is  down.  There  was 
a protracted  period  of  decision-making  which  involved  selection  of  the  detailed 
means  for  implementing  ORTS  goals.  Alternative  data  transmission  methods 
and  devices  were  examined.  Decisions  were  required  in  the  configuration 
and  placement  of  data  collection  devices.  This  detailed  selection  process 
consisted  largely  of  comparing  technical  performance,  risk,  operating 
flexibility,  and  cost  of  alternatives. 


Figure  5-1  is  a simplified  block  diagram  of  the  selected  concept.  The  data 
transmission  method  uses  a data  bus  with  a serial  digital  data  format.  Where 
parallel  data  must  be  sensed,  it  is  collected  in  parallel  and  stored  at  data 
collection  terminals  for  later  serial  transfer.  The  transmission  system 
selection  criteria  included  cost,  accuracy,  ease  of  installation,  noise  immunity, 
error  detection,  and  EMI  generating  considerations.  The  data  collection  inter- 
face selection  was  influenced  by  the  Navy's  MIL-STD-1326  ATE  interface  speci- 
fication. All  analog  signals  are  conditioned  and  normalized  to  a standard  dc 
range  for  conversion  to  the  digital  transmission  format  at  the  data  collection 
terminals.  Digital  signals  are  collected  serially  or  in  parallel,  synchronously 
or  non- synchronously.  The  signal  conditioners  are  designed  as  modules  which 
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Figure  5-1.  Mk  545  Operational  Readiness  Test  System  (ORTS)  Block  Diagram 
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are  interspersed  with  prime  equipment  modules.  The  data  collection  terminals 
are  separate  standard  assemblies  located  in  prime  equipment  cabinets  and 
housing  A/D  data  point  address  recognition  and  transmission  devices.  Data 
from  each  test  point  is  transmitted  on  command  from  the  AEGIS  central  pro- 
cessor. ORTS  utilizes  a shared  input/output  line  from  an  AEGIS  AN/UYK-7 
computer. 

Since  the  ATE  and  prime  equipment  were  developing  together,  an  ATE  configura- 
tion had  to  be  selected  which  could  accommodate  changes  in  the  prime  system. 
Flexibility  and  ease  of  expansion  were,  therefore,  heavily  weighted  evaluation 
factors.  The  data  collection  terminals  are  readily  expanded  or  contracted  to 
accommodate  subsequent  design  changes  or  other  applications,  and  the  data 
interface  was  standardized  early  in  the  program  to  ensure  compatibility  of  ORTS 
and  prime  equipment  designs. 


i.  Another  area  of  decision  was  the  fault  isolation  depth  to  be  selected.  Projected 

design  cost,  technical  risk,  and  complexity  increased  as  fault  isolation  approached 
i . the  single  module  level.  In  the  other  direction  the  logistic  advantages  of  ORTS 

diminished  as  the  number  of  modules  in  a fault-isolated  group  arose.  A com- 
promise was  effected  wherein  a fault  isolatable  group  would  consist  of  an 
average  of  5 modules.  The  eventual  deployment  of  a module  tester  with  opera- 
tional models  will  enable  rapid  isolation  of  the  faulty  module  in  a group. 


5. 1 . 3 Production  Phase  - Hughes  Aircraft  Corp. 


5.1.4  Inventory  Phase  - Hughes  Aircraft  Corp. 
5.2  TRANSCEIVER  - Hughes  Aircraft  Corp. 
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5.3  INTEGRATED  RADIO  ROOM 


5.3.1  Introduction 

The  Integrated  Radio  Room  (IRR)  is  similar  to  the  AEGIS  system  in  its  heter- 
ogeneity, but  considerably  smaller  in  scope.  Concept  design  was  performed 
by  the  Navy,  and  the  stringent  logistic  and  manning  considerations  of  the 
submarine  environment  clearly  pointed  to  an  on-line  automatic  test  system. 

As  its  title  implies,  it  is  an  integrated  communications  center,  and  control  is 
automated  and  centralized  to  the  extent  practicable.  Since  communication 
processing  requirements  could  be  satisfied  only  by  a computer,  the  opportunity 
presented  itself  for  a relatively  sophisticated  Control,  Monitor  and  Test  system 
tCMT),  without  unduly  complicating  the  overall  system,  by  shared  use  of  one  of 
the  communications  control  computers.  This  discussion  will  be  limited  to  the 
Monitor  and  Test  portions  of  that  system,  which  are  the  areas  of  interest  of 
this  report  and  which  do  not  involved  classified  items  in  the  system. 

5.3.2  ATE  Selection  Procedure 


The  Navy  had  already  made  the  generic  ATE  decision  based  on  availability, 
maintainability,  and  manning  constraints,  so  that  the  contractors  (three  were 
selected  for  design,  two  for  prototype  construction)  were  started  at  point  2 of 
Figure  4-1.  The  selection  process  was  therefore  centered  on  means  to  imple- 
ment the  generic  configuration  selected  by  the  Navy.  Widely  different  test 
philosophies  had  to  be  accommodated  because  of  the  fact  that  economics  dictated 
that  existing  developments  be  used  if  technically  acceptable.  The  major  evalua- 
tion factor  was  cost,  and  a very  strong  case  had  to  be  made  for  any  technical 
benefit  which  raised  cost.  This  is  understandable,  because  an  Integrated  Radio 
Room  is  already  basically  more  expensive  to  buy  (although  not  necessarily  to 
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own)  than  the  conventional  radio  room , because  of  the  additional  cost  of  the 
integrating  elements  over  that  of  the  communications  gear  alone. 

Details  of  the  selected  configuration  will  be  omitted  from  this  discussion 
because  the  CMT  is  essentially  a computer-controlled  multiplexed  information 
data  bus  similar  to  the  AEGIS  ORTS.  More  pertinent  to  this  report  was  the 
necessity  to  depart  from  automated  and  centralized  testing  where  the  benefits 
were  not  justified  by  the  added  cost.  This  example  demonstrates  that  formal 
trade-offs  are  not  required  for  decisions  involving  on-line  ATE  when  the 
desired  alternative  is  obvious,  based  on  some  dominant  evaluation  factor  - 
cost,  in  most  cases. 

VLF  Subsystem:  The  Very  Low  Frequency  (VLF)  subsystem  contains  its  own 
signal  processing  equipment  which  includes  sufficient  data  processing  capability 
to  perform  internal  status  monitoring  and  testing.  Allowing  VLF  test  to  be  a 
built-in  function  saved  the  cost  of  the  interface  that  would  have  been  required 
if  test  functions  had  been  externally  implemented.  The  interface  provided  is 
reduced  to  that  necessary  to  enable  overall  control  from  the  CMT  Console  and 
to  transmit  status  information  to  the  CMT  Console.  This  subsystem  is  the 
most  complex  and  it  has  the  highest  utilization  rate  of  any  submarine  communi- 
cation equipment.  It  was  designed  especially  for  the  IRR.  Therefore,  there 
was  a clear-cut  justification  for  built-in  comprehensive  automatic  testing. 

LF  Radios;  The  Low  Frequency  (LF)  radios  are  standard  AN/WRR-3s.  No 
automatic  or  remotely  indicated  test  provisions  are  included.  However,  the 
use  of  two  radios  offers  some  redundancy,  and  the  cost  of  a major  redesign  to 
include  either  built-in  test  or  an  interface  to  permit  external  on-line  testing 
could  not  have  been  justified. 
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HF  Subsystem:  The  High  Frequency  (HF)  Subsystem  includes  three  receiver/ 
exciters,  one  power  amplifier  which  is  connected  to  one  of  the  receiver/exciters, 
and  an  auxiliary  receiver.  All  except  the  auxiliary  receiver  are  components  of 
the  MK  LC  HF  transceiver  system  described  in  Section  3. 1. 3.  The  auxiliary 
receiver  has  no  on-line  test  provisions;  the  others  use  BITE.  Constant  para- 
meters, such  as  power  supply  and  synthesizer  outputs  are  continuously  monitored 
internally  and  on-line,  in  a one-second  automatic  sequence.  When  one  of  those 
parameters  fails,  the  number  of  the  faulty  module  is  indicated.  A sequential 
automatic  test  of  all  modules  can  be  run  by  momentarily  taking  the  receiver/ 
exciter  and  associated  power  amplifier  off-line.  When  a fault  is  detected,  an 
indicator  displays  the  identification  number  of  the  faulty  module.  The  power 
amplifier  is  tested  with  the  receiver/exciter  to  which  it  is  connected.  The 
module  test  can  be  initiated  either  at  the  equipment,  or  by  command  of  the  CMT 
computer.  The  selection  of  this  alternative  was  influenced  by  the  cost  advantage 
and  the  lack  of  technical  risk  in  accepting  a method  which  had  already  been 
designed  and  proved.  The  designers  of  the  HF  equipment  selected  the  BITE 
approach  because  the  availability  internally  of  suitable  stimulus  signals  made 
possible  comprehensive  on-line  fast- operating  testing  which  caused  no  significant 
increase  in  equipment  complexity  and  cost.  An  interface  to  enable  external  on- 
line testing  would  have  been  more  complex  and  costly  than  the  BITE  devices. 

An  off-line  test  set  would  have  increased  MTTR,  and  lowered  availability. 

UHF  Subsystem:  The  Ultra- High  Frequency  (UHF)  subsystem  includes  one 
AN/WSC-3  transceiver.  Like  the  HF  units,  constant  parameters  are  continuously 
and  automatically  monitored,  including  phase  lock  loop  voltage,  oven  operation, 
oscillator  and  power  supply  outputs.  However,  fault  indication  is  provided  by 
a single  light  which  glows  when  any  one  of  these  parameters  is  faulty.  Further 
built-in  tests  are  manually  initiated  and  sequenced  by  a rotary  switch  on  the 
equipment  panel.  Results  are  read  on  a go /no-go  meter.  Cost  was  the 
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dominant  evaluation  criterion  which  led  to  acceptance  of  the  combination  of 
automatic  and  manual  testing  built-in  to  the  AN/WSC-3.  Although  the  test 
approach  is  operationally  less  convenient  than  that  of  the  HF  radios,  availability 
goals  are  adequately  met,  and  a costly  redevelopment  was  avoided. 

Subsystem  Auxiliaries:  Interconnection  of  antennas , multi  couplers,  and 
peripheral  devices  is  computer- controlled,  and  the  interconnections  are  auto- 
matically monitored.  The  alternative  of  manual  patching  and  verification  by 
inspection  was  entirely  incompatible  with  the  concept  of  an  integrated  radio 
room  and  was,  therefore,  eliminated  in  earlier  Navy  studies.  Computers  are 
all  self-tested  by  software. 


i . 

i ' 
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SECTION  6 
APPENDIX1 


6.1  EVALUATION  FACTORS 


This  section  contains  a listing  of  evaluation  factors  for  use  in  comparing  ATE 
alternatives.  It  is  possible  to  develop  evaluation  equations  by  assigning  values 
and  weighting  each  factor.  Weighting  will  depend  on  individual  applications  and 
may  contain  a high  degree  of  subjectivity.  Cost  factors  can  be  used  without  any 
manipulation,  and  rough  benefit/cost  ratios  can  then  be  developed  by  dividing 
the  summation  of  weighted  benefits  by  cost  for  each  alternative.  Evaluation 
factors  may  be  used  individually  for  screening  purposes,  eliminating  ATE 
candidates  for  such  reasons  as  being  too  large  for  the  available  space,  too 
costly  for  the  available  funds,  or  incapable  of  meeting  environmental  require- 
ments. Figure  6 1 summarizes  evaluation  factors  and  the  methodology  for  their 
use.  Evaluation  i;  seen  as  a sometimes  iterative  process,  wherein  compromises 
in  ATE  requirements  may  be  compelled  by  technical  and  fiscal  realities. 

6.1.1  Technical  Factors 


Technical  evaluation  factors  are  useful  for  initial  screening  purposes.  Later  in 
the  evaluation  proce^,  if  otherwise  attractive  candidates  are  identified  which 
do  not  quite  meet  all  technical  evaluation  factors,  then  technical  factors  may  be 
assigned  values  and  weighted  for  use  in  a trade-off  process.  The  high  cost  of 
militarized  equipment,  for  example,  is  resulting  In  serious  consideration  being 
given  to  the  previously  unthinkable  prospect  of  shipboard  installation  of  ATE 

designed  to  best  commercial  standards. 

T 

Parallel  tasks  were  assigned  to  Hughes  Aircraft  Corporation  for  those  described 
in  Sections  6.1  and  6.2, 
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Test  Spectrum:  From  an  analysis  of  prime  equipment  test  requirements,  a 
spectrum  is  generated  of  test  stimuli  and  measurements.  From  this  listing  a 
data  bank  can  be  scanned  for  candidates  in  existing  inventory. 

Environmental:  Although  the  acceptability  for  military  use  of  high  grade  com- 
mercial ATE  is  growing,  there  are  still  applications  which  demand  design  to  the 
more  stringent  MIL  environmental  requirements.  Environmental  specifications 
are,  therefore,  an  important  technical  evaluation  factor. 

Physical  Characteristics:  Space,  weight,  power,  and  cooling  requirements  are 
typical  physical  characteristics  to  be  evaluated. 

A vailability/Risk:  The  extent  to  which  candidates  are  available,  or  the  risks 
of  developing  new  ATE  are  factors  to  be  considered.  There  is  even  a risk  in 
re-procuring  previously  designed  ATEs  if  not  presently  in  production,  in  that 
a significant  start-up  cost  could  be  encountered  or  technological  obsolescence 
could  make  components  difficult  to  procure.  Available  software  versus  the  risk 
of  developing  new  software  must  be  considered  in  view  of  the  length  of  time  and 
cost  to  develop  software. 

6.1.2  Performance  Evaluation  Factors 

Some  performance  evaluation  factors  are  also  technical  in  nature  but  at  a system 
level,  and  are  therefore  treated  separately  from  detailed  technical  evaluation 
factors. 

Performance:  Measures  of  ATE  performance  are  monitoring  and  fault  isolation 
levels,  test  rates,  and  results  formats.  For  some  applications,  the  allocation 
of  on-line  and  off-line  test  tasks  may  also  be  of  significance. 
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Availability:  Availability  evaluation  factors  for  the  ATE  involve  MTBF,  MTTR, 
and  calibration  requirements.  Consideration  is  required  of  the  level  and  speed 
of  self-test,  and  the  relative  degree  of  in-place  calibration  vs.  calibration 
requiring  component  removal. 

Logistics  Factors:  Logistics  factors  are  at  least  as  important  for  the  ATE  as 
for  the  equipment  it  supports,  since  prime  equipment  availability  can  depend 
heavily  on  its  ATE.  Logistic  factors  consist  of  personnel,  training,  spares, 
calibration  facilities,  and  support  documentation.  Operational  and  maintenance 
crew  size  and  skills  need  to  be  identified.  When  tests  will  call  for  operator 
intervention  in  the  test  cycle,  as  often  encountered  with  off-line  IMA  and  depot 
testing,  the  operator  may  need  training  in  UUT  as  well  as  in  ATE  operation. 
Spares  levels,  locations,  and  pipeline  times  are  significant  factors.  The 
location  of  the  calibration  facility  (if  needed)  and  any  special  equipment  for  that 
purpose  should  also  be  identified. 


Growth  Potential  and  Flexibility:  These  factors  break  down  into  excess 
technical  requirements  and  capacity,  and  design  flexibility.  The  evaluator 
must  decide  what  these  factors  are  worth  to  him,  and  to  do  so,  he  has  to 
estimate  future  needs.  Technical  requirements  in  excess  of  presently  known 
needs  may  be  sought  to  handle  additional  UUT  types,  or  in  anticipation  of 
possible  design  changes  to  existing  UTJTs.  Excess  test  rate  capacity  may  be 
desirable  to  allow  for  UUT  ope  rational  MTBF3  which  may  be  lower  than 
predictions,  or  in  anticipation  of  additional  support  requirements  from  other 
causes,  or  simply,  as  a conservative  allowance  for  ATE  or  personnel  perfor- 
mance which  may  not  come  up  to  expectations.  Flexibility  of  design  is  related 
to  excess  technical  requirements.  It  is  desirable  as  a means  for  avoiding  early 
obsolescence,  and  to  meet  supported  system  changes.  Flexibility  of  design 
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will  facilitate  establishing  expanded  or  contracted  configurations  to  serve  a 
variety  of  applications  from  one  design  family. 

Commonality  Factors:  Related  to  logistics  and  flexibility  factors  are  common- 
ality factors.  It  is  clearly  desirable  for  an  ATE  to  use  a minimum  of  newly 
designed  assemblies.  The  ATE  could  use  assemblies  from  another  ATE  or 
from  the  prime  equipment,  as  in  the  so-called  "hot  mock-up"  type  of  special 
test  sets. 

Cost:  Cost  factors  dominate  any  ATE  evaluation,  and  will  continue  to  do 
so.  Costs  go  beyond  the  ATE  itself  and  can  be  significant  where  supported 
equipment  must  be  modified  for  compatibility  with  the  ATE.  It  has  become 
well-known  through  disillusioning  experience  that  software  costs  can  exceed 
hardware  costs  in  some  applications.  As  with  any  other  equipment,  documen- 
tation costs  for  ATE  can  also  be  significant.  Acquisition  costs  and  life  cycle 
costs  may  have  to  be  separately  considered.  Figure  6-2  tabulates  a hierarchy 
of  items  that  make  up  total  ownership  costs,  and  Figure  6-3  separates 
acquisition,  application,  and  usage  costs.  A listing  of  cost  factors  follows: 

ATE  Acquisition, 'Re-acquisition/Modification  Cost:  Is  new’  ATE  to  be  required  ? 
Existing  ATE  to  be  re-acquired?  (Beware  start-up  and  inflationary  escalation.) 
Can  existing  ATE  be  used  or  re-acquired  with  modifications  to  do  the  job9 

UUT/ATE  External  Interface  Hardware:  These  are  the  adapter  boxes  and  cables 
used  between  UUTs  and  ATE.  Although  potentially  a costly  item  (and  a storage 
and  retrieval  problem),  it  is  usually  far  less  costly  than  to  alter  the  UUT  and 
the  ATE  to  eliminate  the  need  for  them. 
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SOFTWARE  MAINTENANCE 
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POWER 

MODIFICATIONS 
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CALIBRATION 
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FACILITY  OVERHEAD 
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SOFTWARE 
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EXECUTIVES 

ROUTINES 

(DESIGN  & VALIDATION) 
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' 

SIMULATORS 
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[ 

PROGRAM  MANUALS 

OPERATOR/MAINTENANCE 

MANUALS 

INTERFACE  DEVICES 
UPDATED  PROGRAM  MANUALS 
OPERATOR/MAINTENANCE 

SELF  TEST  PROGRAMS 

TRAINING 

Figure  6-3.  Resources  Required  - Dollars 

UUT  Modification:  UUT  modification  should  be  examined  with  care.  A simple 
UUT  modification  to  provide  compatibility  with  ATE  can  be  accompanied  by  a 
major  documentation  and  spares  provisioning  change.  This  is  particularly  sig- 
nificant where  only  a portion  of  the  UUTs  will  be  so  modified,  leading  to  a 
possible  nomenclature  change  and  re -qualification. 

Software:  Costs  are  incurred  In  preparation  of  UUT  test  software,  ATE  self- 
test software,  software  preparation  aids  such  as  compilers  and  assemblies, 
and  in  maintenance  of  software  as  UUTs  change. 


ATE  Support:  The  ATE  will  need  support  funding  for  spares,  special  test  or 
calibration  equipment,  and  operational  and  maintenance  documentation. 

Manpower:  Training,  salaries,  and  berthing  costs  of  the  operational  and  main- 
tenance crews  are  a factor. 

Physical:  Costs  for  real  estate  or  ship  space,  power,  and  other  physical  plant 
facilities  should  be  identified. 

6.1.3  Quantification  of  Evaluation  Factors 

in  order  to  minimize  the  subjectivity  that  is  inevitably  part  of  any  equipment 
trade-off,  attempts  should  be  made  to  assign  numerical  values  to  evaluation 
factors.  Certain  factors  are  normally  expressed  in  numbers,  and  they  should 
be  used  as  suen  with  weighting  constants  used  to  match  particular  applications. 
Factors  which  are  normally  expressed  numerically  are: 

Costs 

Testing  rate,  UUT  and  self 

Availability 

MTI3F 

MTTR 

Maintenance  man  hours  per  operating  hour 

Operating  man  hours  per  operating  hour 

Space 

Weight 

Power 

Environmental  specifications 
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Figure  6-4  evaluates  in  steps  the  acceptable  level  of  risk  on  a basis  of 
i operational  need  date  vs.  ATE  design  and  production  values.  At  one  extreme, 
a six-month  operational  need  date  calls  for  selection  of  an  ATE  which  is 

I 

Ik  already  operational  and  with  free  time  to  handle  the  proposed  application.  At 
the  other  end  of  the  scale,  only  an  operational  need  date  more  than  two  years 
away  can  accommodate  a new  ATE  design  as  an  acceptable  risk. 


* . Other  factors  do  not  lend  themselves  well  to  quantification  with  any  degree  of 
mathematical  rigor.  A case  in  point  is  "growth  potential  and  flexibility"  one 
».  of  the  suggested  performance  evaluators.  Suppose  that  the  candidate  ATE 

stimulus  spectrum  exceeds  present  requirements.  How  should  the  excess  be 

I 

• • evaluated? 


j 

Dividing  the  candidate  by  the  required  spectrum  gives  a factor  of  two,  for  the 
case  of  10  Hz  - 200  kHz  divided  by  2 Hz  - 100  kHz.  Would  a ratio  of  octaves 
1 * be  more  significant?  Then,  too,  is  the  2 Hz  limit  more  or  less  valuable  than 

the  extension  from  100  to  200  kHz  for  the  foreseeable  future  of  the  particular 

i . HARDWARE 
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EQUIPMENT  IN  OPERATION,  FREE  TIME  AVAILABLE 
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LOCATION 


EQUIPMENT  IN  SERVICE  TEST,  EXPECT  PRODUCTION 
~ - - _ WITHIN  ONE  YEAR 
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- - _ ___NEW  DESIGN 

Figure  6-4.  Risk 
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prime  equipment  under  consideration?  A temptingly  easy  way  out  is  to  use  a 
binary  approach,  with  yes  or  no  for  the  existence  or  absence  of  a factor,  but 
obviously  this  leads  to  being  able  to  beat  the  system  by  providing  trivial 
advantages  which  will  score  high  but  possibly  be  of  little  benefit.  Regardless 
of  the  obvious  difficulties,  an  attempt  should  be  made  to  attach  numerical  values 
to  all  evaluation  factors,  using  whatever  criteria  seem  suitable.  The  values 
and  their  weighting  coefficients  may  be  subjective  and  very  much  application- 
dependent,  but  the  approach  will  be  more  useful  than  one  which  has  no 
numerical  base.  Fortunately,  cost,  which  is  the  most  significant  evaluation 
factor,  is  numerical  in  nature.  The  evaluator's  only  problem  with  costs  is 
their  credibility,  and  this  can  be  handled  by  dividing  estimated  costs  by  their 
confidence  factor  to  give  a factored  cost. 


6. 2 CHECK  LIST 


This  is  a list  of  items  which  should  be  cons  id.  red  in  the  course  of  the  ATE 
selection  and  evaluation  procedure.  The  list  includes  procedural  steps, 
evaluation  factors  and  prerequisite  information. 

Obtain  prime  equipment  descriptive  material 
Obtain  support  concept  description 
Develop  test  requirements  listing 

Stimulus 

Measurement 

Switching 

On-line/off-linc  limitations 
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Examine  prime  equipment  UUTs 

Accessibility 
Test  interface 

Identify  external  interface,  if  required 
Identify  costs 

ATE  hardware 
ATE  self-test  software 
Software  aids,  compilers 
UUT  test  software 

UUT  external  interface  devices,  adapters 
ATE  operator  manning  and  training 
ATE  maintenance  manning  and  training 
ATE  spares 
ATE  facilities 
Software  maintenance 

UUT  modifications,  including  support  impact 
Documentation 

Analyze  evaluation  factors 

Test  spectrum 
Environmental  specs 
Physical  characteristics 
Design/availability  status 
Risk 

Monitoring  level 
Fault  isolation  level 
On-line/off-line  capability 
Displays 
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Test  rate 
MT13E 
Ml  Til 
Self-test 

Calibration  requirements 

Personnel  skills 

Training 

Spares  policy 

Documentation 

Growth  potential  'flexibility 

Commonality  to  existing  equipment 


fi.  3 TEST  EQUIPMENT  TYPES 


6.3.1  Introduction 


This  section  provides  basic  definitions  of  test  equipment  terms  and  a brief 
survey  of  test  equipment  types.  The  information  is  not  intended  for  systems 
engineers  assigned  to  ATE  selection  who  will  already  be  familiar  with  the  field, 
but  for  the  familiarization  of  others  involved  in  the  prime  equipment  acquisition 
process  whose  background  may  be  in  other  disciplines. 


6.3.2  Basic  Terms 


Terms  defined  and  discussed  in  this  section  have  been  used  to  describe  test 
equipment  elements,  configurations,  and  operational  modes.  Confusion  some- 
times results  where  the  same  terms  are  used  to  define  operational  modes  and 
equipment  configurations,  and  where  one  configuration  can  operate  in  more  than 
one  mode  fe.g. , on-line,  off-line).  It  will  also  be  apparent  from  the  listing 
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below  that  there  is  considerable  overlap  in  terms.  No  attempt  is  made  here 
to  provide  an  extensive  glossary.  MII.-STD- 1309,  Definition  of  Terms  for 
Automatic  Test  and  Checkout,  performs  that  function.  The  definitions  belojv 
are  limited  to  the  major  terms  used  in  this  report,  plus  some  encountered  in 
test  equipment  literature  that  are  not  significant  enough  or  were  created  too  late 
to  have  been  included  in  MIL-STD-1309,  but  which  could  confuse  anyone  unfamiliar 
with  their  intent.  Terms  are  arranged  alphabetically. 

Automatic:  Automatic  Test  Equipment  (A.TE)  or  automatic  test  systems  (ATS) 
are  those  which  automatically  perform  test  sequences  and  logical  operations  to 
determine  operational  status  or  corrective  maintenance  action  required.  Auto- 
matically controlled  through  a computer  or  a tape  sequencer  are  stimulus  gen- 
erator selection  and  setting,  measurement  device  selection  and  ranging,  con- 
nection sequencing  to  the  unit  under  test  (UUT),  and  display  or  recording  of  the 
conclusions  derived  from  the  test  logic.  The  attainable  degree  of  automaticity 
is  limited  by  the  test  accessibility  of  the  UUT.  Particularly  at  depot  levels, 
it  is  often  necessary  for  the  ATE  operator  to  disassemble  the  UUT  to  enable 
manual  connection  to  test  points  during  the  otherwise  automatic  sequence.  An 
ATE  can  be  designed  for  optional  manual  operation  where  maintenance  policy 
and  level  of  operator  training  permit  manual  operation.  Automatic  test  systems 
can  be  configured  using  building  blocks  which  are  conventionally  designed  as 
Conventional,  Manual,  or  Standard  (which  see).  So-called  third  generation  ATE 
systems,  now  growing  in  prominence,  depart  somewhat  from  the  direct 
association  of  test  functions  with  building  blocks,  in  that  the  computer, 
formerly  used  only  as  a control  and  calculating  device,  now  also  performs  some 
of  the  stimulus  and  measurement  functions. 

Ruilt-In:  Built  *n  test  (BIT)  or  built-in  tost  equipment  (BITE)  are  test  devices 
which  are  an  integral  part  of  the  equipment  being  tested.  BIT  may  be  automatic, 
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manual,  on-line,  off-line,  or  a combination  thereof.  Although  BIT  and  BITE 
are  often  used  interchangeably,  BIT  may  more  correctly  be  considered  to  be 
the  function  performed  by  BITE.  Also  see  Dedicated. 

Centralized:  A centralized  test  system  is  one  which  processes,  at  a central 
location,  information  gathered  by  test  point  data  sensors  at  more  than  one 
remotely  located  equipment  or  system  under  test.  (See  Federated  and  Dedicated.) 
Centralized  test  systems  are  usually  automatic,  although  a manual  system 
could  be  envisaged  which  is  also  centralized.  An  extensive  installation  is 
implied,  such  as  one  which  would  test  a ship's  communication  system,  or  other 
major  '‘quipment  grouping. 

Conventional:  Conventional  describes  general  purpose  or  standard  (which  see) 
laboratory  type  test  equipment,  such  as  signal  generators  and  digital  voltmeters. 

Dedicated;  A dedicated  test  equipment  (or  system)  is  one  which  is  always 
connected  to  a particular  test  specimen.  The  definition  differs  from  built-in,  in 
that,  while  a dedicated  test  set  may  be  co-located  and  even  housed  with  equipment 
it  is  to  test,  it  could  conceivably  be  removed  intact  for  use  elsewhere.  When  a 
dedicated  test  system  is  used  for  more  than  one  UUT,  it  becomes  Centralized  or 
Federated.  This  is  clearly  a confusing  term,  made  worse  by  the  occasional 
usage  of  BITE  for  Dedicated.  Fortunately,  Dedicated  is  more  often  used  to 
categorize  ATE  computers,  rather  than  entire  ATE  systems. 

Fault  Detection:  This  is  the  process  of  detecting  the  existence  of  a fault  without 
actually  locating  it.  It  is  thus  similar  to  Readiness  Testing  and  Performance 
Monitoring,  which  see. 
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which  is  manually  operated.  The  operator  of  manual  test  equipment  sets  all 
stimulus  and  measurement  devices  and  connects  them  to  test  points  in  the  unit 
being  tested.  He  also  reads  individual  test  results,  evaluates  performance, 
and  diagnoses  faults. 

Measurement:  The  meaning  is  obvious.  Voltmeters,  ammeters,  frequency 
counters,  pressure  gauges,  and  thermometers  are  examples  of  measurement 
devices. 

Monitoring/Pcrfonnancc  Monitoring:  Monitoring  usually  signifies  merely 
reading  test  parameters  at  intervals  (or  continuously)  without  the  further 
application  of  test  logic  for  interpretation  of  those  readings.  Two  diverse 
examples,  monitoring  Exhaust  Gas  Temperature  of  a gas  turbine  engine,  or 
reading  catho:le  current  of  a linear  amplifier  do  not  in  themselves  necessarily 
indicate  performance  or  condition.  Adding  an  interpret"  ! ion  function  through 
test  logic  converts  Monitoring  into  Performance  Monitoring,  which  is  simply 
another  name  for  testing.  The  distinction  between  monitoring  and  testing  seems 
especially  artificial  when  it  is  considered  that  monitoring  of  transmitter  power 
output,  for  example,  is  a means  of  testing  the  performance  of  that  unit  with  no 
further  logic  required  except  to  verify  that  the  transmitter  is  turned  on. 

Performance  Monitoring  is  best  defined  as  describing  a test  system  operational 
mode  in  which  tests  are  limited  to  those  necessary  to  determine  whether  specified 
overall  performance  is  being  attained.  Similarly  to  Fault  Detection,  the  causes 
for  malfunction  arc  not  necessarily  identified.  Performance  Monitoring  also 
bears  similarities  to  Readiness  testing  (which  see).  Although  Performance 
Monitoring  is  usually  On-Line,  the  term  does  not  exclude  Off-Line  application. 
Performance  Monitoring  can  lie  applied  to  a test  equipment  category  as  well  as 
an  operational  mode. 
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Fault  Isolation  (Level):  Fault  Isolation  is  the  operating  mode  in  which  the 
automatic  test  system  identifies  the  source  of  a malfunction  within  the  equipment 
or  system  which  it  is  testing.  The  level  to  which  fault  isolation  is  achieved 
refers  to  the  equipment  generation  breakdown  level,  in  descending  order: 
group  of  modules  (or  assemblies);  module;  component.  The  two  upper  levels 
are  also  called  Weapon  Replaceable  Assembly  (WRA)  and  Shop  Replaceable 
Assembly  (SRA). 

Fault  Location  (Level):  Same  as  Fault  Isolation  (Level). 

Federated:  A little  used  term,  Federated  is  a variation  of  the  centralized 
system,  and  implies  a similarly  extensive  installation.  The  idealized  cen- 
tralized system  operates  exclusively  from  raw  data  obtained  from  test  point 
sensors  in  each  equipment  to  be  tested,  and  does  all  processing  centrally.  The 
federated  concept  is  similar,  except  that  it  denotes  acceptance  of  pre-processed 
results  data  from  BITE  or  dedicated  test  equipment  associated  with  the  equip- 
ment to  be  tested. 

Integrated:  This  is  a term  recently  adopted  by  the  Navy  to  describe  an  auto- 
matic test  system  (LATS)  which  is  connected  to  a major  system  or  a number 
of  sh4p  systems  and  equipments.  Integrated  implies  less  rigidity  regarding 
central  data  processing  than  was  assigned  to  Centralized,  and  is  thus  closer 
to  Federated  in  meaning,  but  clearer.  It  is  not  yet  certain  whether  Integrated 
will  grow  into  a generic  term,  or  whether  it  will  be  associated  only  with  the 
first  system  of  its  type,  originally  conceived  for  the  Trident  program. 

Manual:  Manual,  is  often  used,  as  synonymous  with  Conventional  or  Standard 
(which  see)  laboratory  test  equipment,  and  is  opposite  to  Automatic.  However, 
it  can  also  refer  to  General  or  Special  Purpose  (which  see)  test  equipment 
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Off-  Line /On-  Line:  These  terms  often  cause  confusion  because  they  are  used 
to  describe  test  equipment  configurations  as  well  as  test  equipment  operating 
modes,  and  the  two  categories  are,  more  often  than  not,  inconsistent. 
MIL-STD-1309  is  clear  in  its  definition  of  off-line  and  on-line  testing.  Off-line 
testing  consists  of  tests  performed  while  the  tested  equipment  is  not  performing 
its  normal  operational  function.  A radio  receiver  which  is  temporarily  discon- 
nected from  its  antenna  to  receive  a test  stimulus  is  considered  as  being  tested 
off-line,  even  if  the  test  is  being  performed  at  the  operational  site.  Any  oper- 
ational interruption  during  a test  qualifies  that  test  as  off-line.  The  most 
obvious  example  of  off-line  testing  is  the  removal  of  an  equipment  from  its 
installation  site  for  test  at  another  location.  An  on-line  test  is  one  which  is 
performed  while  the  equipment  is  in  normal  operation,  and  which  in  no  way 
degrades  or  interferes  with  the  normal  operational  use  of  the  supported 
equipment.  A simple  example  would  be  a voltage  measurement  and  evaluation 
device  which  indicated  when  a power  supply  was  operating  out  of  tolerance,  but 
which  did  not  affect  the  performance  of  the  supply  in  its  operational  environ- 
ment and  usage. 

Off-line  test  equipment  is  equipment  which  is  not  permanently  connected  to  the 
supported  equipment.  A depot  test  set  is  off-line.  Shop  test  equipment  is  off- 
line if  it  is  only  connected  to  the  supported  equipment  during  corrective  or 
periodic  maintenance.  On-line  test  equipment  is  always  connected  to  the 
supported  equipment.  BITE  is  on-line.  Also,  any  external  test  equipment, 
whether  centralized,  dedicated,  or  any  other  type,  becomes  on-line  if  it  is 
always  connected  to  the  prime  equipment. 


It  is  clear  that  both  off-line  and  on-line  testing  can  be  performed  by  either 
off-line  or  on-line  test  equipment,  and  to  avoid  confusion,  the  terms  on-line 
and  off-line  must  be  qualified  as  to  whether  they  refer  to  test  mode  or  to  test 
equipment. 

Purpose,  General/Multi-/Special;  Terms  which  describe  Purpose  are  imprecise 
and  overlap  enough  to  generate  considerable  confusion.  They  should  be  used 
sparingly. 

General  Purpose:  General  Purpose  test  equipment  can  be  automatic  or  manual 
although  it  usually  relates  to  the  latter.  General  Purpose  usually  means  that 
capabilities  and  operational  flexibility  exist  beyond  those  immediately  needed, 
thereby  enabling  a wide  range  of  UUT  types  to  be  tested.  For  example,  a 
frequency  synthesizer  may  be  automatically  programmable  (or  manually  settable) 
from  1-50  MHz,  even  though  only  a portion  of  that  range  may  be  needed  by  the 
group  of  UUTs  for  which  the  synthesizer  was  originally  designed.  General 
Purpose  can  appear  to  imply  universal  applicability,  which  is  not  necessarily  so. 

Multi- Purpose:  Multi-Purpose,  as  the  name  implies,  merely  signifies  that  the 
test  equipment  can  be  used  for  more  than  one  UUT.  Although  the  term  is  not 
much  used,  it  is  more  precise  than  General  Purpose,  and  interchangeable  with 
it,  depending  on  the  subjective  evaluation  of  when  the  number  of  purposes 
approaches  general  as  a limit. 

Special  Purpose:  Special  Purpose  is  most  accurately  defined  in  a negative  sense. 
It  denotes  test  equipment  designed  for  a particular  purpose  and  probably  inade- 
quate for  any  other  purpose.  Special  purpose  test  equipment  will  employ  spot 
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frequency  generators  instead  of  wide  range  signal  generators,  and  switching 
and  logic  (perhaps  wired-in)  will  be  tailored  exclusively  for  the  one  application. 

Readiness  Testing,  Operational/System;  Except  for  fine  shadings  of  meaning, 
Operational  and  System  Readiness  testing  are  identical  to  Performance 
Monitoring.  Usage  to  date  implies  that  Readiness  testing  is  done  an  a larger 
scale  than  Performance  Monitoring.  For  example,  the  AEGIS  Operational 
Readiness  Test  System  (ORTS)  determines  readiness  of  an  entire  weapon 
system,  including  data  processors,  radars,  and  missile  systems.  The 
distinction  between  Operational  and  System  Readiness  becomes  cloudy,  except 
where  operational  readiness  is  defined  as  requiring  readiness  of  more  than  one 
system.  In  the  other  direction,  if  performance  of  an  HF  transmitter  is  moni- 
tored as  satisfactory,  the  radio  man  would  consider  that  unit  to  be  operationally 
ready. 

Standard:  Standard  test  equipment  describes  normal  laboratory  test  equipment, 
usually  already  in  Inventory,  or  of  a similar  type. 

Stimulus:  MIL-STD-1309  succinctly  states  it:  "Stimulus  is  any  physical  or 
electrical  force  applied  to  a device  intended  to  produce  a measurable  response." 
Audio  and  radio  frequency  signal  generators  are  stimulus  devices.  Power 
supplies  to  power  the  unit  being  tested  are  classifed  as  stimulus.  Even  ihough 
passive,  a dummy  load  is  also  considered  to  be  stimulus. 

Switching:  The  connection  or  re-connection  of  stimulus  and  measurement  devices 
are  referred  to  as  switching  in  automatic  testing,  where  the  implementation  can 
be  a significant  part  of  the  system.  The  term  is  not  used  as  much  in  normal 
testing,  where  switching  is  an  inherent  part  of  using  stimulus  and  measurement 
equipment. 
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Unit  Under  Test  (UUT):  The  most  important,  despite  its  alphabetical  ranking, 
is  the  unit  under  test.  MIL-STD-1309:  "Any  system,  set,  subsystem,  assembly, 
subassembly,  and  so  forth,  undergoing  testing."  Past  literature  may  contain 
references  to  AUT  or  SUT,  for  assembly  or  system  under  test,  respectively. 

The  MIL-STD-1309  definition  is  comprehensive  enough  to  make  the  others 
unnecessary. 

6.4  DATA  BANK  Hughes  Aircraft  Corporation 

Although  shown  as  a part  of  Section  6 for  the  sake  of  completeness,  the  Data 
Bank  results  are  so  extensive  that  a separate  document  will  probably  result. 


Procedure  for  the  Selection 
of 

Automatic  Test  Equipment  (ATE) 


August  1974 


i 

I 

I 

l 


ATEMAT  (MAT  03T) 

HEADQUARTERS,  NAVAL  MATERIAL  COMMAND 


4-t 


PROCEDURE  FOR  THE  SELECTION 
OF 

AUTOMATIC  TEST  EQUIPMENT  (ATE) 


AUGUST  1974 


ATEMAT  (MAT  03T) 

Headquarters,  Naval  Material  Command 


'***t»*»»»,««Kfl (W’WWWft 


EXECUTIVE  SUMMARY 


SITUATION 

As  a function  of  its  responsibility  in  the  ATE  (Automatic  Test  Equipment) 
area  the  ATE  Management  and  Technology  Office  (ATEMAT  MAT  03T)  conducted  a 
review  of  ATE  selection  and  acquisition  procedures  within  the  Naval  Material 
Command  and  surfaced  a major  problem  In  the  time  phasing  of  ATE  selection  and 
acquisition  events.  To  alleviate  the  problem  ATEMAT  sponsored  the  development 
of  a selection  procedure  emphasizing  proper  timing.  Follow-on  efforts  will 
address  the  acquisition  procedures  as  a natural  augmentation  to  this  procedure. 

METHODOLOGY 

Investigation  by  the  Test  Equipment  Technology  Support  Office  (TETSO)  of 
the  support  activity  (including  Integrated  Logistics  Support  and  its  components) 
having  a direct  impact  on  the  ATE  selection  process  revealed  that  essential 
actions  were  often  not  accomplished  during  the  proper  time  in  a development 
phase  and  in  some  cases  in  later  phases.  A joint  task  group  comprised  of  ATE, 
logistics  support,  and  modeling  specialists  was  assembled  at  NELC  to  review 
previous  study  results  in  ATE  selection  and  to  identify  the  areas  that  needed 
improvement.  Review  of  current  ATE  selection  practices,  related  support  activ- 
ities, and  pertinent  documentation  was  conducted  by  the  joint  task  group. 

Tools,  aids  and  the  phasing  guidelines  for  ATE  selection  events  were  developed 
and  incorporated  in  this  report. 

CONCLUSIONS 

The  most  important  conclusions  drawn  from  this  study  (and  documented  in 
this  report)  are: 

a.  Selection  of  ATE  is  an  integral  part  of  prime  system/platform  design. 

b.  The  Advanced  Development/Validation  phase  Is  the  most  critical  phase 
in  the  ATE  selection  process. 

c.  Selection  of  ATE  is  an  integral  part  of  the  ILS/LSA  process. 
RECOMMENDATIONS 

This  selection  procedure  should  be  Incorporated  Into  an  ATE  selection  and 
acquisition  guide,  and  other  documents  pertaining  to  the  selection  process 
should  be  modified  to  reflect  adherence  to  these  recommended  procedures,  thtll  ihe 
selection  and  acquisition  guide  is  available,  this  document  should  be  used  as 
an  interim  guide  to  the  selection  of  ATE. 
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1.0 


INTRODUCTION 


1.1  Background 

The  selection  of  the  proper  automatic  teat  equipment  (ATE)  to  satisfy  a 
platloriu  or  prime  system  need,  has  proved  to  be  a complex  task  which  Is  often 
underestimated.  Various  concepts  and  hardware  must  be  proposed,  and  each 
compared  with  one  another  in  relation  to  their  performance,  availability, 
utilization  and  the  affect  on  the  operational  readiness  of  that  platform  or 
system.  In  the  past,  mistakes  have  been  made  which  have  caused  delays  in 
deployment,  excessive  costs,  poor  reliability  and  inadequate  performance.  This 
report  describes  procedures  to  be  employed  in  the  selection  of  the  proper  ATE 
for  a given  job.  The  framework  is  built  around  the  acquisition  process  as 
described  in  SECNAVINST  5000.1  and  the  Integrated  Logistic  Support  (ILS)  pro- 
cedures identified  in  NAVMATINST  4000.20. 

1.2  Applicable  Documents 

These  documents  apply  generally  to  the  subject  of  the  procedure.  For 
references  which  contain  more  details,  consult  the  documents  listed  in  Section  7. 

a.  NAVMATINST  3960. 4A  of  26  December  1973;  this  instruction  provides 
policy  and  responsibility  for  automatic  test,  monitoring  and  diagnostic  systems 
and  equipment. 

b.  NAVMATINST  4000. 20A  of  18  March  1971;  integrated  logistic  support 
planning  policy. 

c.  MIL-STD-1388  of  15  October  1973;  this  military  standard  describes  the 
logistic  support  analysis  procedure. 

d.  SECNAVINST  5000.1;  all  Navy  system  acquisitions  are  covered  by  this 
instruction. 

e.  OPNAVINST  5000.42  of  1 Jun  1974;  Weapons  systems  selection  and  planning 
is  revised  by  this  instruction. 

1.3  Purpose 

The  procedures  set  forth  herein  are  to  be  used  by  Program  Managers, 
Acquisition  Managers  and  others  associated  with  the  acquisition  of  Navy  hardware. 
Since  each  program  is  unique,  this  procedure  has  been  structured  for  selective 
use,  allowing  the  user  to  select  guidance  responsive  to  the  needs  of  his  program. 
The  procedure  is  divided  into  five  sectionswhich  parallel  the  phases  of  the 
acquisition  process  set  forth  in  SECNA7INST  5000.1.  Each  section  provides  the 
guidance  needed  during  that  phase  of  a hardware  (i.e.  platform,  system,  equipment) 
acquisition  that  will  insure  concurrent  development  and  acquisition  of  associated 
ATE. 


Selection  of  ATE  is  a support  function  that  is  a part  of  the  Integrated 
Logistic  Support/Logistic  Support  Analysis  (ILS/LSA)  areas  of  platform  and  prime 
system  development.  All  of  the  actions  taking  place  in  parallel  to  the  ATE 
selection  are  included  for  each  phase  of  the  acquisition  cycle.  This  enables 
the  planners /managers  to  key  the  ATE  selection  actions  to  events  in  the  acquisition 
process.  Too  often  ILS  actions  in  general,  and  ATE  selection  in  particular, 


takes  place  later  than  what  would  be  optimum.  Only  by  the  parallel  development 
of  ATE  and  the  prime  equipment,  can  it  be  assumed  that  the  equipment  will  be 
fully  effective  when  Introduced  for  fleet  use. 

1.4  Approach 

Each  section  of  this  procedure  sets  forth  an  objc.  tive  to  be  achieved,  in 
so  far  as  ATE  is  concerned,  during  a single  phase  of  equipment  acquisition.  Step- 
by-step  procedures  are  provided  for  achieving  this  objective,  assuring  con- 
sideration is  given  to  each  aspect  of  the  problem  and  that  these  actions  are 
taken  in  the  proper  sequence.  To  further  assist  in  the  acquisition  process, 
existing  aids  to  assist  in  this  process  are  Identified,  along  with  their 
recommended  use. 

Input  requirements  for  each  phase  are  identified,  and  to  the  extent 
possible,  the  sources  of  these  inputs  are  named.  Outputs  or  products  that 
result  from  the  satisfactory  accomplishment  of  a phase  are  also  defined  to  further 
assist  in  identifying  the  tasks  to  be  accomplished  during  each  phase  of  the 
acquisition. 

1.5  Organization  of  Report 

This  report  is  composed  of  four  general  areas  which  consist  of  Section  1, 
Sections  2 through  6,  Section  7 and  Section  8.  These  cover  the  following 
areas: 


Section  1 - Provides  Introduction  and  a management  overview. 

Section  2 - 6 - Contains  a detailed  description  of  the  major  efforts  per- 
formed during  each  prime  hardware  development  phase.  Each  phase  is  covered  in 
a separate  section. 

Section  7 - Is  a list  of  the  tools  and  aids  that  are  available  to  the 
Program  Manager  and  his  supporting  1LS  managers,  to  assist  him  in  the  ATE 
Selection  Process. 

Section  8 - Provides  conclusions  and  recommendations  for  follow-on  efforts. 

1.6  Overview  or  ATE  Selection  Procedure 


During  the  Conceptual  Phase,  the  prime  hardware  is  defined  basically  by 
needs  and  objectives.  General  support  requirements  are  specified,  such  as 
operational  availability.  From  these,  an  ATE  concept  can  be  defined  in  terms 
of  broad  performance  monitoring  needs  and  the  degree  of  off-line  testing 
needed  at  each  maintenance  level. 

Advanced  Development  Phase  activities  involve  definition  of  the  support 
system  in  relation  to  the  hardware  design.  On-line  ATE  (BIT/BITE)  hardware 
must  be  designed  in  parallel  with  prime  equipment  design.  This  phase  ends 
with  the  contract  to  proceed  with  the  detail  design  and  the  Full  Scale  Develop- 
ment Phase. 


In  addition  to  the  prime  hardware  design  and  development,  the  support 
equipment  (including  ATE)  is  also  developed  during  the  Full  Scale  Development 
Phase.  Prototype  ATE  is  evaluated  and  assessed.  Redes ign/modiflcatlon  is  also 
performed. 

During  the  Production/Construction  Phase,  off-line  ATE  specifications  are 
completed,  and  procurement  initiated  and  completed.  Software  and  Test  Program 
Sets  (TPS)  are  developed  for  support/test  of  the  prime  hardware.  Changes  and 
modifications  to  the  ATE  hardware/software  may  result  from  the  TPS  production. 
Evaluation  and  assessment  of  the  ATE  should  also  take  place  during  this 
operation.  Information  gained  from  the  evaluation/assessment,  if  of  sufficient 
magnitude,  can  serve  as  a foundation  for  a major  Engineering  Change  Proposal 
(ECP)  to  clean  up  any  deficiencies  in  the  ATE. 

1.6.1  Selection  Process 


Figure  1-1,  ATE  Selection/Prime  Hardware  Acquisition  Process,  provides  a 
flow  chart  description  of  this  overview.  This  figure  forms  the  basis  of  sub- 
sequent sections.  However,  it  should  be  recognized  that  (1)  the  process  is 
iterative  in  nature  and  (2)  the  ATE  selection  process  must  be  tailored  to 
specific  prime  hardware  peculiarities. 

1.6.2  Phase  Flow  Overview 


There  are  four  major  prime  hardware  development  phases  which  are  covered 
in  Figure  1-1  for  a brief  overview  of  the  ATE  selection  process. 

Figure  1-1  is  a matrix  of  these  four  major  development  phases  against  the 
following: 


a.  Objectives 

b.  Principal  Efforts 

c.  Logistic  Support  Factors 

d.  Logistic  Support  Design  Process 

e.  ATE  Selection  Process 

f.  Output  Documentation 

Each  of  the  following  five  sections  is  organized  using  (a)  through  (f) 
categories  as  headings.  Categories  (a)  and  (b)  are  overall  prime  hardware  system 
design  functions.  Categories  (c)  and  (d)  are  ILS  functions.  Category  (e)  is  the 
ATE  selection  process  and  category  (f)  summarizes  the  output  documentation 
relative  to  the  entire  process. 
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FIGURE  11.  ATE  SELECTION/PRIME  HARDWARE  ACQUISITION 
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1.7  Personnel  Engaged  In  the  Selection  Process 

Selection  of  the  proper  ATE  Is  normally  a team  effort.  Three  types  of 
personnel  are  usually  required  on  the  selection  team. 

a.  A representative  from  the  prime  hardware  design  team,  especially  when 
BIT/BITE  Is  usdd. 

b.  A representative  from  the  ILS  manager's  office. 

c.  An  expert  on  the  design  and  application  of  ATE. 

The  first  two  categories  are  normally  a part  of  the  prime  hardware  development 
team.  Expertise  on  ATE  selection  can  be  obtained  from  a variety  of  sources. 

The  most  appropriate  starting  points  are  the  SYSCOM  ATE  focal  points  (AIR  5342, 
ELEX  4804  or  SEA  98)  or  ATEMAT  (MAT  03T)  and  the  Test  Equipment  Technical  Sup- 
port Office  (NELC  4050). 


2.0  CONCEPTUAL  PHASE 


2.1  Objectives 

During  the  conceptual  phase  of  development,  the  prime  objectives  are  to: 

a.  Reveal  whetner  or  not  there  are  similar  or  duplicate  developments 
underway  which  meet  the  operational  requ  1 remen t**.  Information  resulting  from 
this  analysis  could  either  provide  a system  cor..ept  for  further  development,  or 
eliminate  the  proposed  development  because  of  parallel  development  of  another 
system  that  satisfies  the  requirements; 

b.  Select  a preferred  system  concept;  and 

c.  Determine  whether  the  preferred  system  Is  sufficiently  attractive  to 
warrant  movement  to  the  advanced  development  phase. 

Note  that  these  obectives  relate  to  the  total  conceptual  phase  activity 
of  which  the  ATE  selection  process  is  a subset. 

2.2  Principal  Efforts 

Principal  efforts  during  the  conceptual  pha^e  are: 

a.  Identification  and  definition  of  conceptual  systems. 

b.  Analyses  (i.e.,  cost,  threat,  mission,  feasibility,  risk,  trade-offs, 
logistic  support,  and  worth)  of  the  alternatives. 

c.  Design,  experimentation  and  test  of  operational  requirement  assumptions 
and  marginal  technology.  Most  of  the  activity  during  the  conceptual  phase  of 

the  system  development  cycle  Is  not  hardware  oriented,  but  some  hardware  related 
effort  may  De  required.  This  Involves  testing  items  of  hardware  which  represent 
advances  in  technology  (or  on  the  borderline  of  new  technology).  Other  experi- 
mentation may  involve  simulating  (with  mock-ups)  operational  situations  to 
establish  their  feasibility. 

d.  Highly  iterative  activities  generating  data  supporting  continuation  of 
the  program  Into  the  Val  idation  or  Advanced  Development  phase. 

Part  of  the  identification  of  concqjtual  systems  (2.2a)  Involves  the 
Identification  of  "ATE  concepts"  in  addition  to  the  other  prime  system  concepts. 

The  same  activities  that  apply  to  the  prime  system  conceptualization  procedure 
also  apply  to  the  ATE  associated  with  it.  These  activities  Include  analysis  and 
trade-offs,  contributing  to  the  decision  process  for  further  development. 

2.3  Logistic  Support  Factors 

In  the  conceptual  phase,  the  following  planning  and  other  activities  should 
take  pi ac^ : 

a.  Initial  ILS  planning  of  a generic  type. 
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b.  Functional  analysis  to  develop  recommended  support  alternatives  for  [ 

appropriate  logistic  elements  (the  ten  components  of  ILS). 

c.  Meeting  the  need  for  a broad,  general  integrated  logistic  support  f 

plan,  and  noting  any  special  problems  that  have  been  observed  through  the  * 

logistic  effort 

i . 4 Logistic  Support  Design  Process 

Most  of  the  following  material  was  developed  around  the  Military  Standard 
on  Logistic  Support  Analysis  (MIL-STD-1388)  and  is  presented  in  a form  that  Is 
suitable  for  ATE  selection  discussion.  Refer  to  Figure  1-1,  Conceptual  Phase, 

Logistic  Support  Design  Process  to  follow  this  discussion. 

The  two  inputs  to  this  process  are  the  operational  requirements  and  the 
historical  data.  These  are  indicated  by  the  shaded  boxes  on  the  left  hand  side 
of  Figure  1-1  (in  Logistic  Support  Design  Process).  Operational  Requirements 
have  four  components: 

Mission  Requirements 

Deployment  Requirements 

Functional  Requirements 

Performance  Requirements 

The  first  two  of  these  requirements  lead  to  use  studies  which  yield 
Mission  Profiles,  Utilization,  Deployment,  Environment,  Physical,  and  Maintenance 
Constraints. 

Historical  data  results  in  desirable  design  concepts  and  state-of-the-art 
design  objectives. 

Functional  design  studies  are  conducted  on  the  output  data  from  the  Opera- 
tional Requirements  and  historical  data  paths.  These  functional  design  studies 
involve  describing  each  function  as  a "black  box"  with  an  input  and  output. 

Each  functional  box  is  placed  in  the  design  and  the  following  factors  are  deter- 
mined: 


Reliabil ity/Maintainabi  1 ity  Estimates 

Design  Objectives 

Performance  Characteristics 

Design  Constraints 

Functional  Allocation  (i.e.,  partitioning  the  functions  among  various 
building  blocks) 

It  is  during  the  functional  design  studies  that  various  approaches  to  ATt, 
and  BIT  (Built-In  Test)  should  be  considered.  Remember  its  never  too  early  to 
start  planning! 

From  the  Functional  Design  Study  output,  the  Support  Synthesis  Process  can 
start.  The  results  of  the  support  synthesis  process  are: 

Support  System  Structure 
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MBS  (Work  Breakdown  Structure)  for  the  support  system 

Logistic  Constraints 

Functional  Requirements  (Identification  and  refinement  of  the  initial 
functional  requirements— this  process  continues  throughout  development) 

MOE/FOH  (Measure  of  Effectiveness/ Figure  of  Merit)  Development 

Special  Requirements  (this  is  another  point  at  which  special  purpose 
ATE  or  BIT/BITE  can  be  considered) 

It  is  after  these  operations  that  the  Logistic  Support  Requirements 
Identification  takes  place.  Logistic  Support  Requirements  Identification  starts 
during  the  Conceptual  Phase  and  continues  through  the  full  scale  development  phase. 
One  of  the  things  which  are  accomplished  during  the  identification  of  logistic 
support  requirements  is  the  functional  definition  of  ATE/BIT.  The  major  events 
In  this  identification  process  are: 

a.  Maintenance  Plan  - consisting  of: 

* Maintenance  Concept  (general) 

' Environmental  and  Mission  Constraints 

* LOR  (Level  of  Repair)  Concepts/ Constraints 

b.  Support  Equipment: 

Functional  Requirements  for  all  support  equipment  including  General 
Purpose  Electronic  Test  Equipment  (GPETE),  ATE  and  BIT 

c.  Supply  Support: 

Supply  Concepts  are  Identified  to  support  the  prime  system  - the 
prime  system  supply  concept  will  have  a direct  Impact  on  ATE  selection,  for  it 
may  dictate  where  repair  and  test  of  a given  Item  will  take  place. 

d.  PHST  (Packaging,  Handling,  Storage  and  Transportation): 

Transportation  Policies  are  established 
Constraints  are  identified  and  placed  on  PHST 

e.  Facilities: 

. Special  needs  are  identified 

f.  Personnel/Training: 

. Manning  Standards  for  the  prime  equipment  and  support  system  (ATE 
Included)  are  identified 

. Special  Training  requirements  are  surfaced  - this  includes  ATE 
operation  and  support  (software  and  programing  is  germane) 

2.5  ATE  Selection  Process 

2.5.1  General 

Definition  of  the  ATE  concept  Is  the  main  thrust  of  the  Conceptual  Phase 
of  the  ATE  Selection  Process.  There  are  two  main  functions  which  must  be  performed 
in  the  ATE  concept  formulation  phase.  These  are  establishing  performance  moni- 
toring need  and  determining  the  degree  of  off-line  ATE  at  organizational 
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intermediate,  and  depot  levels.  As  indicated  in  Figure  1-1,  there  are  three 
subsets  of  the  Performance  Monitoring  Needs  that  need  to  be  considered. 

a.  To  what  level  will  system  and  platform  performance  monitoring  be 
performed: 


b.  What  environmental  factors  (i.e.,  safety,  damage  control,  EMCON-Electo- 
magnetic  Emission  Control)  will  be  monitored? 

c.  What  system  configurations  need  to  be  displayed  (i.e.,  communication 
channels,  electrical  power  plant  status)? 

2.5.2  Selection  Procedure 

Procedures  which  are  followed  in  the  ATE  selection  process  are  linked  to 
various  steps  of  the  Logistic  Support  Design  Process.  An  orderly  step  by  step 
general  ATE  selection  process  would  follow  the  order  presented  in  Figure  2-1, 
Preliminary  Support  Concept  Definition.  Each  of  the  following  paragraphs  are 
keyed  to  the  steps  outlined  in  Figure  2-1. 

2.5. 2.1  Define  Hardware 

It  is  during  the  hardware  definition  cycle  of  the  system  to  be  supported, 
that  the  ATE  concept  is  based.  Preliminary  hardware  definition  efforts 
correspond  to  the  Functional  Design  and  Support  Synthesis  activities  in  the  Logistic 
Support  Design  Process.  The  factors  which  are  considered  in  the  hardware 
definition  are  performance  requirements,  a functional  description  of  the  total 
system  (or  platform)  and  its  building  blocks,  reliability,  maintainability, 
availability,  budgetary  factors  (cost,  space,  weight,  and  power),  and  hardware/ 
module  breakdown.  Since  during  the  conceptual  stage  the  direct  association 
with  hardware  is  limited  to  similar  systems,  or  known  components  which  will 
become  a part  of  the  prime  hardware;  little  information . is  available  on  input/ 
output  parameters,  operating  tolerances,  and  detailed  hardware  layouts/wiring  dia- 
grams . 


From  the  information  available  at  this  stage  of  development,  it  is  possible 
to  start  formulating  the  performance  monitoring  needs  of  the  system.  Such 
things  as  BIT,  BITE,  On-Line  Test,  and  Self  Test  needs  can  be  formulated.  These 
needs  can  then  be  blended  into  the  definition  of  the  prime  system  and  the  support 
system.  This  effort  will  produce  a preliminary  integrated  prime  hardware  and 
support  system  design.  Subsequent  iterations  with  the  other  factors/operators 
in  the  definition  process  may  change  the  hardware  definition  but  certain  general 
attributes  remain  constant.  It  is  quite  feasible  to  define  BIT/BITE  or  On-Line 
Test  during  the  first  cut  at  a hardware  definition  (design)  end  have  it  remain 
relatively  stable  through  the  development  cycle.  Because  of  the  highly  inte- 
grated nature  of  BIl/BITE  and  the  prime  hardware  it  is  almost  mandatory  that  it 
be  specified/defined  during  the  conceptual  phase  of  development.  It  is  too 
late  to  wait  until  the  full  scale  development  or  production  phases  to  introduce 
BIT/BITE  or  On-Line  ATE.  Performance  monitoring  needs  should  also  be  identified 
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Figure  2-1  Preliminary  Support  Concept  Definition 


early  in  the  development  cycle  so  that  the  requisite  sensors  and  monitoring 
points  can  be  designed/ built  into  the  prime  hardware.  The  decision  to  use  full 
or  partial  off-line  ATE  can  also  be  made  this  early  in  the  development  cycle, 
and  they  should  be  in  order  to  a. low  an  evolutionary  selection  process.  (See  2. 5.2. 3) 

2. 5. 2. 2 Postulate  Alternative  ILS  Concepts 

Alternative  ILS  concept  formulation  is  a process  which  involves  Level  of  Repair 
(LOR)  concept/constraint  trade-offs,  and  support  system  structure.  A variety  of 
iterations  and  the  trade-offs  are  possible  with  impact  on  the  full  range  of  ATE 
concepts.  This  trade-off  of  alternative  concepts  is  a factor  in  the  definition 
of  ATE  alternatives. 

2. 5. 2. 3 Define  ATE  Alternatives 

For  each  alternative  maintenance  concept  there  may  be  more  than  one  test 
equipment  alternative.  The  scope  of  ATE  alternatives  is  greater  during  the 
conceptual  phase  than  it  is  at  later  development  phases.  Following  are  the 
steps  taken  in  the  definition  of  alternatives. 

a.  Propose  Generic  ATE  Types  Generic  ATE  types  are  proposed  which  are 
compatible  with  each  maintenance  concept  under  study  as  a first  step  in  the 
selection  process.  Possible  generic  ATE  options  for  a system  can  be  made  up 
from  one  or  a combination  of  the  following: 

. Built-In  Test  (BIT) 

. Built-In  Test  Equipment  (BITE) 

. Other  on-line  test  systems 

. Off-line  test  systems 

b.  Identify  On-Line  Test  Requirements  Prime  system  and  mission  needs  will 
have  to  be  analyzed  to  determine  the  need  for  on-line  monitoring.  In  general, 
from  an  operational  and  maintenance  viewpoint,  on-line  monitoring  (or  testing)  is 
the  most  desirable  mode  of  operation.  The  trade-offs  involved  are  cost,  technical 
impact  on  the  design,  and  the  operational  requirements  for  the  mission. 

c.  Identify  the  Degree  of  Off-line  Test  Requirements  Many  of  the  selection 
considerations  for  on-line  test  requirements  are  also  applicable  to  off-line 
testing.  The  degree  of  off-line  testing  will  largely  depend  on  decisions  regarding 
specific  maintenance  levels  and  locations  (i.e.,  organizational,  intermediate  and 
depot) . 

2. 5. 2. 4 Select  Best  ATE  Alternative 

At  the  conceptual  level  of  development  the  ATE  alternatives  can  only  be 
matched  with  the  degree  of  detail  available  on  the  prime  hardware,  its  cost 
estimate,  and  the  technical  definition  of  detail  of  the  hardware  alternatives. 

It  is  quite  likely  that  several  possible  acceptable  alternative  ATE  concepts  will 
still  exist  after  the  end  of  the  concept  phase.  Further  refinement  of  the  ATE 
optimization  process  should  take  place  during  later  stages  in  the  system  develop- 
ment such  as  during  Advanced  Development  and  Full  Scale  Development. 
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2.!j.2.5  Select  ILS  Concept 

Selection  of  the  maintenance  and  ILS  concept  will  determine  the  ATE  concept 
selection. 

2. 5.2.6  Define  Support  Concept 

As  a result  of  the  above  procedures,  the  ILS  concept  and  ATE  concept  have 
been  selected.  From  these  a support  concept  can  be  defined  which  will  provide  a 
clear  understanding  for  both  the  ILS  and  the  ATE  personnel. 

2.6  Output  Documentation 

There  are  several  documents  which  result  from  the  conceptual  phase  of  develop- 
ment. These  are  listed  here  In  order  to  maintain  the  frame  of  reference  for 
the  cognizant  ATE  specialist  to  relate  his  ATE  procedure  to  other  ILS  and  prime 
hardware  development  actions.  The  following  documents  (see  Figure  1-1)  are  out- 
puts of  the  conceptual  phase: 

a.  DP  (Development  Proposal) 

b.  PMP  (Program  Master  Plan) 

c.  APP  (Advanced  Procurement  Plan) 

d.  DCP  (Development  Concept  Paper) 

The  DP  documents  the  decisions  made  on  the  selection  of  ATE,  and  is  therefore 
the  most  relevant  piece  of  output  documentation. 


3.0  ADVANCED  DEVELOPMENT  PHASE 


In  the  advanced  development  or  validation  phase,  continuation  of  the  prime 
hardware  development  takes  place.  Advanced  development  activities  for  the  prime 
hardware  support  system  and  ATE  all  proceed  in  parallel.  Support,  1LS,  main- 
tenance planning,  LSA  and  prime  hardware  design  actions  take  place  as  an  inte- 
grated process  and  tradeoffs  are  made  between  them  to  produce  an  Advanced 
Development  i*lodel  (ADM)  and  support  system.  The  material  covered  in  this  section 
is  keyed  to  Figure  1-1.  It  is  germane  to  note  that  the  advanced  development 
phase  represents  the  transition  from  the  high  risk  to  low  risk  area  of  develop- 
ment. 


3.1  Objectives 

There  are  five  main  oojectives  for  the  advanced  development  phase  listed  in 
Figure  1-1  winch  highlight  the  goals  of  this  phase  of  development.  Each  of 
these  apply  to  the  selection  of  ATE  as  well  as  the  remainder  of  the  prime  system 
or  platform  development.  The  item  which  is  significant  to  the  ATE  objectives 
is  the  establishment  of  firm  and  realistic  performance  specifications,  including 
technical  interfaces.  It  is  important  that  performance  specifications  and  hard- 
ware interfaces  be  established  for  the  ATE  and  all  prime  system/ platform  areas 
which  have  any  impact  on  the  ATE. 

3.2  Principal  Efforts 

Four  main  areas  of  activities  are  listed  in  Figure  1-1  for  the  principal 
efforts  during  the  advanced  development  phase.  In  each  of  the  areas  listed 
in  the  development  flow  chart  the  activity  followed  for  the  prime  hardware  also 
pertains  to  the  ATE  and  support  system.  Prototype  hardware  development  of  the 
prime  system  as  well  as  the  ATE  should  be  accomplished.  Further  refinement  of 
performance  specifications,  and  validation  of  conceptual  studies  which  cover  the 
ATE/BIT  and  prime  hardware  should  be  made.  A preliminary  engineering  design  for 
the  total  system/platform  should  result  from  the  advanced  development  phase 
efforts.  One  of  the  main  efforts  is  to  reduce  the  degree  of  risk  associated 
with  both  the  ATE  and  prime  hardware  development. 

3.3  Logistic  Support  Factors 

There  are  two  important  logistic  support  factors  which  should  be  addressed 
during  the  advanced  development  phase.  Continuing  examination  and  evaluation 
of  support  alternatives,  and  examination  of  special  problems  of  logistic  require 
ments  are  the  two  factors  wnich  should  be  considered.  These  efforts  are  a 
continuation  of  the  effort  that  was  started  during  the  conceptual  stage  of 
development. 

3.4  Logistic  Support  Design  Process 

In  the  corresponding  section  of  Figure  1-1  the  logistic  support  design 
process  is  outlined  in  the  block  flow  chart/diagram  format.  It  is  indicated 
In  the  diagram  that  this  is  a straight  flow  process  (one  way)  leading  through 
to  tne  development  of  the  ADM.  Many  of  the  functions  are  interactive  with.n 
their  own  process  and  with  other  functions.  For  the  sake  of  clarity,  the 
feedback  paths  are  not  shown.  As  an  example  of  the  iterative  or  interactive 


nature  of  a function,  consider  the  Support  System  Definition  function  block.  In 
the  process  of  performing  the  Support  System  Definition  function  support  system 
alternatives  are  balanced  against  factors  su-h  as  support  facility  locations 
and  transportation  requirements.  LOR  and  otk*>-  analyses  are  made  which  are 
optimizing  processes  (cost  is  the  main  optimization  factor).  The  results  of 
these  intra-function  analyses  are  then  balanced  with  othe»  external  functions 
to  achieve  a total  optimized  system.  For  example,  In  the  LOR  process  It  could 
be  possible  to  determine  the  lowest  cost  maintenance/repair  system  for  prime 
hardware  with  the  requirement  for  a maintenance  depot(s)  that  does  not  exist. 
Examination  of  the  external  world  would  reveal  that  no  suitable  depot(s) 
exist  and  the  optimum  LOR  system  could  call  for  the  establishment  of  depots 
at  a great  initial  outlay  of  construction  and  other  high  startup  funds. 

3.5  ATE  Selection  Process 


3.5.1  General 

Postulation  of  ATE  alternatives  and  selection  of  the  "best"  or  optimum 
approach  is  the  central  theme  of  ATE  selection  during  the  advanced  development 
phase.  This  process  involves  the  two  categories  outlined  in  Figure  1-1  as 
follows: 

a.  On-Line  ATE  (includes  BIT/BITE) 

b.  Off-Line  ATE 

Each  of  these  functions  is  discussed  in  the  following  paragraphs. 

3.5.2  Selection  Procedure 

In  the  conceptual  phase,  as  outlined  in  Section  2,  the  ATE  concept  (or 
several  equal  concepts)  will  be  developed.  From  this  ATE  concept  (or  concepts) 
alternative  approaches  will  be  determined,  examined,  and  a selection  of  the 
best  alternative  made.  Alternative  systems  are  matched  with  the  associated 
proposed  support  approach,  and  the  selected  system  will  combine  the  best  ATE 
alternative  with  it. 

3.5.2. 1 On-Line  ATE 

a.  Research.  Appropriate  data  banks  and  literature  searches  should  be 
made  to  Identify  possible  testing  concepts  and  techniques.  This  is  a difficult 
area  to  catalog  and  index,  especially  for  BIT/BITE.  The  SETE  data  bank  listed 
in  Section  7,  is  available  for  searches  of  this  type. 

b.  Analysis.  Design  approaches  identified  during  the  research  procedure 
that  were  applied  to  similar  systems  or  platforms  should  be  analyzed  for  their 
applicability  to  the  prime  system  being  developed.  The  term  "design  approach" 
is  more  important  to  the  on-line  test  selection/design  because  BIT/BITE  must 

be  inherently  designed  into  the  prime  hardware  rather  than  selecting  it  off-the- 
shelf.  It  is  expected  that  techniques  used  for  the  same  class  of  platforms 
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should  be  transferable  to  new  platforms  and  its  associated  support  electronics. 
Identification  of  a computer/simulation  design  aid,  such  as  the  NORDEN  BITE 
Model,  cited  in  Section  7,  serve  to  help  formalize  the  BIT/BITE  design  process 
and  optimize  results. 

c.  Selection  of  Best  Approach.  From  the  background  research  and  analysis, 
the  "best  approach"  can  be  selected  after  applying  the  total  system  life 

cycle  cost;  considering  the  effect  on  the  platform  or  system  operational 
readiness;  and  technical  risk  and  schedule.  The  tools  listed  in  Section  7,  can 
aid  in  this  evaluation.  One  of  the  prime  objectives  of  the  advanced  development 
phase  are  to  reduce  the  technical  risks  Involved  in  system  development. 

d.  Design.  Once  the  best  on-line  test  approach  has  been  selected  with 
associated  support  design  tools,  the  BIT/BiTE  can  be  designed  into  the  prime 
system  or  platform. 

3.5. 2. 2 Off-Line  ATE 

In  the  case  of  off-line  ATE  selection,  the  process  lends  itself  to  the 
Identification  of  specific  off-the-shelf  ATE  or  ATE  "building  blocks".  Follow- 
ing are  the  events/actions  which  should  take  place  to  select  off-line  ATE. 

a.  Search.  A search  of  ATE  data  banks  and  literature  should  be  made  to 
match  Unit  Under  Test  (UUT)  requirements  with  available  inventory  and  other 
acceptable  off-line  ATE.  Data  banks  are  particularly  well  suited  to  the  match- 
ing of  off-line  ATE  to  UUT  requirements.  The  ATE  data  bank  (Avionics  Systems 
Test  Equipment  Comparator,  ASTEC)  at  NAEC  can  provide  a very  accurate  (on  a 

pin  by  pin  basis)  match  between  the  UUT  and  ATE  in  their  data  bank.  In  addition, 
abbreviated  summary  data  on  the  ATE  is  also  available  to  users.  ASTEC  will 
also  file  the  UUT  data  as  part  of  its  data  bank  so  that  future  runs  can  be 
made  against  the  UUTs.  At  SAALC  (operational  2nd  quarter  FY  75)  their  ATE  data 
bank  will  match  UUT  requirements  against  ATE  and  ATE  building  blocks.  Summary 
data  requirement  inputs  are  described  for  each  UUT  to  be  screened  against  the 
ATE  in  the  SAALC  data  bank.  Use  of  these  data  banks  can  be  identified  through 
the  SYSCOM  ATE  contacts  or  ATEMAT/TETSO. 

After  the  data  bank  search  and  other  research  has  taken  place,  the  results  can 
be  analyzed  and  alternatives  can  be  identified.  Iterations  in  this  process  can 
take  place  if  insufficient  data  is  accumulated  during  the  first  pass. 

b.  Identify  Alternatives.  Wnen  the  data  banks  have  been  searched  and 
other  research  and  analysis  has  bien  completed,  alternative  off-line  ATE  can 
be  identified,  and  also  alternative  approaches  can  be  determined.  Alternative 
ATE  (or  building  blocks)  can  be  identified  from  existing  inventory  assets,  or 
determined  to  be  available  off-the-shelf.  When  the  existing  ATE  capability  does 
not  meet  the  needs  for  the  UUT  test  capability,  modification  to  the  existing  ATE 
can  be  considered  as  an  alternative.  This  decision  point  can  be  reached  when 
the  search  of  the  data  banks  and  literature  does  not  reveal  any  ATE  which  meets 
a high  percentage  of  the  UUT  test  requirements,  but  shows  a 60  to  80  percent 
coverage.  A percentage  of  UUT  test  capability  is  one  of  the  automated  data 
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bank  search  byproducts.  In  the  event  that  none  of  the  above  approaches  yield  a 
good  solution,  the  final  alternative  is  development  of  new  ATE.  This  is  the 
least  desirable  alternative  due  to  long  lead  times  and  development  costs.  But 
if  new  ATE  must  be  developed  then  the  advanced  development  phase  is  the  point 
in  time  to  start  (not  the  production  phase  where  it  often  happens). 

c.  Evaluation  of  Alternatives.  Alternatives  should  be  weighed  against 
life  cycle  costs  for  the  total  system.  Each  alternative  should  also  be  evaluated 
in  reference  to  the  effect  upon  the  prime  system  or  platforms  operational 
readiness.  Another  factor  which  encompasses  all  types  of  ATE  is  performance. 
Performance  takes  into  account,  probability  of  failure  detection,  fault  isolation 
level  (within  a specified  number  of  modules),  speed,  and  operator  usability 
(consideration  of  human  factors).  Reliability,  availability,  and  logistic  support 
of  the  ATE  itself  are  important  factors  to  be  taken  into  account  in  resolving 
the  selection  of  alternatives.  Utilization  factors  (growth  potential,  installation 
location,  and  excess  available  capacity)  should  be  considered  in  selecting  from 
off-line  ATE  alternatives.  A number  of  tools  and  aids  are  described  in  Section  7 
to  assist  in  these  evaluations. 

In  selecting  alternatives  to  meet  the  constraints  mentioned  above,  quite 
often  the  most  important  one  is  the  development  delivery  schedule.  It  may  be 
required  that  ATE  alternativ.s  may  be  selected  so  that  the  schedule  can  be  met. 
nlhere  a far  superior  approach  has  to  be  abandoned  in  favor  to  one  of  lesser 
acceptability  because  of  schedule  requirements,  it  should  be  reviewed  thoroughly 
to  determine  if  the  schedule  can  stand  this  slippage. 

3.6  Output  Documentation 

At  the  completion  of  the  advanced  development  phase,  the  documents  listed 
in  Figure  1-1  could  result.  Only  the  new  (different  from  conceptual  phase) 
documents  are  discussed  below. 

a.  Preliminary  ILS  Plan.  At  the  conclusion  of  the  advanced  development, 
an  ILS  Plan  should  exist.  It  will  serve  as  the  basis  for  the  ILS  planning  for 
all  following  phases. 

b.  FSD  Specification.  The  results  of  the  advanced  development  phase  and 
evaluation  of  the  ADM  should  include  a Full  Scale  Development  (FSU)  system 
specification  Including  the  specification  for  the  ATE. 

The  initial  LSAR  will  be  prepared.  This  documents  the  Logistic  Support 
Analysis,  including  the  trade-offs  relevant  to  ATE  selection. 
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4.0  FULL  SCALE  DEVELOPMENT  PHASE 

During  the  full  scale  development  phase, the  system/pl attorn  requirement 
and  the  threat  Is  reaffirmed;  for  prime  hardware  and  ATE,  cost  of  ownership  Is 
verified,  practicality  of  the  engineering  design  Is  established,  production  and 
logistics  problems  are  identified  and  considered(solved),  technical  risks  are 
resolved,  operational  suitability  Is  determined  through  test  and  evaluation,  and 
a realistic  plan  for  production  is  developed.  All  of  the  descriptive  material 
presented  in  section  4 is  keyed  to  the  full  scale  development  time  phasing  in 
Figure  1-1.  This  is  the  last  chance  to  resolve  all  the  known  areas  of  risk 
associated  with  both  the  prime  equipment  and  ATE  prior  to  the  production  phase. 

4.1  Objectives 

The  single  major  objective  for  the  full  scale  development  phase  is  to  produce 
a supportable  prime  hardware  model, and  ATE  as  a precondition  to  production  for 
Inventory  or  operational  deployment.  A detailed  system  design,  complete  speci- 
fications, and  associated  support  documents  are  also  central  objectives. 

4.2  Principal  Efforts 

Under  full  scale  development  in  Figure  1-1,  there  are  six  main  efforts  out- 
lined. A continuing  assessment  of  risk  is  required  to  surface  technical  and 
engineering  problems  requiring  study  and  resolution.  Development  milestones,  and  a 
management  system  to  track  them,  should  be  established  to  meet  the  objective. 

All  problems  determined  during  the  validation/advanced  development  phase  should 
be  solved  or  the  risks  reduced  to  an  acceptable  level.  All  data  requirements 
basic  to  the  hardware  design  disclosure  package,  should  be  verified  and  deficiencies 
corrected.  All  logistic  support  requirements  should  be  identified,  and  a formal 
logistic  support  system  should  be  designed  early  in  the  full  scale  development 
phase.  Finally,  completion  of  the  design  of  both  the  prime  hardware  and  ATE 
should  take  place  through  the  full  scale  development  efforts. 

4.3  Logistic  Support  Factors 

In  the  full  scale  development  phase,  a formal  structured  ILS  Plan  (based  on 
preliminary  ILS  Plan)  should  be  developed.  This  is  the  plan  which  will  be  used 
to  support  the  prime  system/platform  during  its  operational  phase.  It  will  also 
outline  the  events  and  actions  which  must  take  place  during  the  production  phase 
to  support  the  operational  system  (including  ATE).  In  addition  to  the  ILS  plan, 
the  complete  logistic  support  program  should  be  reviewed  for  adequacy  to  support 
the  prime  equipment/ platform.  ATE  should  be  considered  in  the  formal  logistic 
support  program  from  both  the  standpoint  of  support  to  the  prime  hardware  and 
support  of  the  ATE  system. 

4.4  Logistic  Support  Design  Process 

For  full  scale  development,  the  logistic  support  design  process  involves 
less  functional  activities  than  the  advanced  development  design  process.  This 
can  be  gleaned  from  the  section  covering  the  subject  in  Figure  1-1.  Signi- 
ficant functions  which  are  implemented  are— the  final  prime  hardware  design. 
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support  system  design  and  user  test/evaluation  of  the  full  scale  development 
model  (FSDM).  In  the  FSDM  evaluation  process,  the  ATE  and  other  support  factors 
will  also  be  evaluated.  The  feedback  from  this  evaluation  will  provide  engineer- 
ing change  and  design  Information  which  can  be  Inputs  to  the  production  specifi- 
cations for  the  ATE  and  prime  equipment.  In  the  logistics  support  requirements 
Identification  area,  the  detailed  maintenance  plan  will  be  completed,  LOR  will 
be  optimized,  support  equipment  (ATE)  requirements,  costs,  and  test  design  will 
be  completed. 

4.5  ATE  Selection  Process 


4.5.1  General 

Selection  procedures  for  both  on-line  ATE  and  off-line  ATE  during  full  scale 
development  is  a process  of  evaluation,  reevaluation  and  assessment  of  the  final 
selection.  Host  of  the  ATE  effort  during  full  scale  development  Involves  the 
design  and  development  aspects  with  very  little  emphasis  on  the  selection  process. 

4.5.2  Selection  Procedure 

Heavy  selection  activity  could  take  place  during  full  scale  development  if 
the  advanced  development  phase  is  skipped  or  abbreviated.  In  that  situation  some 
of  the  selection  activity  will  be  covered  in  the  same  way  as  outlined  in 
paragraph  3.5.2.  The  result  will  be  a compression  of  both  advanced  development 
and  full  scale  development  procedures  into  a single  phase.  This  type  of  program 
phasing  should  only  be  taken  when  many  of  the  unknowns  are  minimal  and  risk  factors 
are  concomitantly  low.  The  tools  and  aids  described  in  section  7 are  again  available 
and  applicable  during  FSD. 

4. 5. 2.1  On-Eine  ATE  (BIT/BITE) 

Following  are  the  actions  which  should  take  place  regarding  on-line  ATE: 

a.  Reevaluation  and  Redesign.  This  includes  considering  the  BiT/BiTE 
design  that  was  completed  during  the  advanced  development  phase  and  refining  it 
from  the  ADM  test  and  evaluation  results.  This  selection  activity  will  involve 

the  selection  of  the  optimum  features  for  the  FSDM  BIT/BITE  design  and  incorporating 
the  desired  features  into  the  FSDM. 

b.  Develop  BIT/BITE.  From  the  design  that  evolved  from  the  reevaluation, 
redesign,  and  refinement  process  BIT/BITE  will  be  built  into  the  FSDM  for  the 
prime  system/platform.  As  part  of  the  selection  process,  this  development  of  a 
FSDM,  with  the  inherent  BIT/BITE  built  into  it,  will  provide  an  evaluation  tool 
for  the  integrated  system/platform.  The  advanced  development  phase  will  provide 
the  last  reasonable  point  where  this  integrated  BiT/BiTE  design  can  be  implemented 
and  evaluated. 

c.  Technical  Evaluation/Operational  Evaluation.  As  a subset  of  the  prime 
system/platform  evaluation,  the  ATE  should  also  be  evaluated  in  the  process. 

This  is  very  significant  in  the  case  of  BiT/BiTE  since  It  cannot  be  evaluated 
without  the  benefit  of  the  prime  hardware  being  in  a fully  developed  status. 
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For  off-line  ATE  it  is  possible  to  perform  a successful  evaluation  with  the 
UUT's  to  be  processed  by  the  ATE.  In  the  on- line  ATE  situation  only  after  the 
BIT/BITE  has  been  integrated  into  the  prime  equipment,  can  it  be  evaluated.  The 
technical  and  operational  evaluations  will  identify  the  merits  of  the  FSDM  ATE 
as  well  as  the  weaknesses.  This  evaluation  should  provide  sufficient  information 
to  assess  the  ATE  for  suitability  to  go  into  production  and  deployment  phases. 

d.  Assessment.  When  the  technical  and  operational  phases  have  been 
completed,  then  the  ATE  selection  can  be  assessed.  This  assessment  can  be  made 
as  a measure  of  the  suitability  of  the  ATE  selected  design  approach  against  its 
technical /operational  evaluation  results.  The  assessment  process  can  be 
quantized  In  terms  of  the  percentage  of  operational  and  technical  objectives 
which  were  met  by  the  FSDM.  Other  assessment  approaches  could  involve  how  well 
faults  were  isolated  and  the  meeting  of  MTTR  specifications  (or  bettering  tne 
requirements).  Since  the  scope  of  the  BIT/BITE  is  fairly  well  committed  at  this 
stage  of  development,  it  is  not  easy  to  measurably  impact  its  performance,  if 
the  negative  aspects  of  the  assessment  are  unacceptable.  An  added  support 
technique,  could  be  considered;  if  for  example,  fault  isolation  to  a single 
individual  module  is  not  attainable,  off-line  screening  ATE  could  be  added  to 
the  development  program  to  resolve  ambiguities  between  modules. 

4.S.2.2  Off-Line  ATE 

Selection  and  evaluation  of  off-line  ATE  follows  the  steps  outlined  in 
Figure  1-1  and  parallels  the  on-line  case  with  the  exception  that  there  is  more 
emphasis  on  software. 

a.  Reevaluation  and  Development/Procurement.  In  light  of  the  full  scale 
development  operation,  the  requirements  for  the  off-line  ATE  should  be  reevaluated 
and  modified  accordingly.  The  experience  from  the  ADM  and  FSDM  evaluations 
should  provide  a strong  input  to  the  requirements  reevaluation.  After  reaffirma- 
tion and  updating  of  the  requirement,  the  off-line  ATE  can  be  developed  or  pro- 
cured. The  selection  procedure  is  identical  to  that  outlined  under  section  3 

(3. 5. 2. 2)  and  should  be  followed  again.  In  addition  to  the  ATE  hardware,  soft- 
ware (operating  system/executive)  must  also  be  selected,  developed  and  procured. 
Test  programs  must  be  developed. 

b.  Evaluate  Programming  Aids.  Support  software  to  assist  in  the 
generation  of  test  patterns,  and  test  programs  should  be  evaluated  against  the 
test  program  requirements.  For  complex  digital  UUT's  (over  50  IC's)  it 
becomes  difficult  to  generate  comprehensive  accurate  test  patterns/programs 
through  mannual  techniques.  An  automatic  te:t  pattern/program  generator  such  as 
U-LASAR  (Oigi test  version-kogic  Stimulus  and  Response)  should  be  considered  for 
complex  digital  logic. 

c.  Technical /Operational  Evaluation.  In  support  of  the  FSDM  evaluation, 
the  off-line  ATE  should  be  evaluated.  As  stated  previously,  off-line  ATE  can  be 
evaluated  as  soon  as  the  UUT's  are  available.  The  total  prime  system  FSDM,  need 
not  be  fabricated  in  a fully  operational  test  bed  in  order  for  evaluation  to 
ppoceed.  In  fact,  it  is  recommended  that  at  the  onset  of  availability  of  UUT's 


that  the  off-line  ATE  technical  evaluation  start.  Operational  evaluation  may 
follow.  If  the  off-line  ATE  has  been  scheduled  for  an  intermediate  or  depot 
type  support  plan,  then  it  will  be  evaluated  separately  from  the  same  physical 
proximity  of  the  prime  system/platform. 

d.  Assessment.  An  assessment  of  the  off-line  ATE  selection  should 
be  performed  to  establish  its  adequacy  to  test  the  specified  UuT's.  This 
assessment  can  be  in  terms  of  meeting  the  MTTR  for  the  prime  system  and  the  MTTR 
for  the  UUT's.  The  number  of  specified  UUT's  to  be  tested,  and  the  number  of 
UUT's  which  cao  be  tested  by  the  FSDM  off-line  ATE  can  also  provide  a measure 
for  assessment  of  the  selection.  Feedback  of  shortfalls  and  deficiencies  can 
provide  an  input  to  ECP  (Engineering  Change  Proposal)  action  to  correct  the 
situation  in  the  FSDM  equipment  or  the  production  specifications  can  be  the 
correction  vehicle. 

4.6  OUTPUT  DOCUMENTATION 

Full  scale  development  activities  will  be  concluded  with  the  primary  end 
product  being  a completed  detailed  design.  Final  ILS  Plan,  and  Production 
Specification.  Other  germane  documents  are  outlined  in  Figure  1-1,  and  are  the 
same  which  applied  to  the  advanced  development  stage. 
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5.0  PRODUCTION  PHASE 

Culmination  of  all  of  the  results  of  the  post  development  phase 
takes  place  during  production  of  the  prime  system/ platform,  including  on/offline 
ATE.  Minimal  ATE  selection  activity  is  expected  to  take  place  during  this  phase 
with  the  emphasis  placed  upon  production  and  procurement.  Much  lif  not  all) 
of  the  risk  should  have  been  resolved  by  this  time.  The  operational  suitability 
of  the  prime  system/ pi  at  form  should  have  been  established,  and  a firm  plan  for 
the  production/procurement  should  exist. 

5.1  OBJECI IVES 

A succinct  statement  of  the  two  main  objectives  of  the  production  phase  Is 
outlined  in  Figure  1-1.  These  objectives  Involve  production/construction  of  hard- 
ware which  would  meet  all  the  requirements,  whether  those  originally  specified, 
or  derived  through  the  development  process.  Risks  should  have  been  eliminated, 
and  the  production  model  (ATE  included)  should  be  a result  of  the  hardware/ 
software  testing  and  redesign  process,  with  appropriate  tradeoffs.  The  resultant 
equipment/ system  should  be  available,  reliable  and  supportable  to  meet  its 
mission  requirements  for  the  duration  of  its  life  cycle. 

5.2  PRINCIPAL  EFFORTS 

I here  are  two  principal  areas  of  effort  (indicated  in  Figure  1-1)  that  are 
planned  in  the  production  phase.  Production  of  the  prime  system  or  platform 
(including  the  support  of  ATE)  is  the  main  thrust  of  the  production  phase.  For 
some  areas  of  activity,  there  is  an  overlap  effort  for  items  not  completed  during 
the  full  scale  development  phase.  These  overlap  areas  are  due  to  the  existence  of 
long  lead  time  production  items  and  the  need  to  complete  fragments  of  the  full 
scale  development  process.  Overlap  from  the  full  scale  development  efforts  are 
possible  due  to  evaluation  of  the  FSDM  and  its  spinoff  activities. 

5.3  LOGISTIC  SUPPORT  FACTORS 

Five  logistic  support  factors  are  covered  in  Figure  1-1  for  the  production 
phase.  A validated  or  completed  LSAR  should  result  from  the  completion  of  the 
production  phase  and  it  will  also  be  the  primary  output  documentation.  Timely 
delivery  of  logistic  support  resources,  Including  the  ATE,  is  an  important 
logistic  support  factor  which  should  be  monitored.  Any  variation  in  the  delivery 
schedule  for  the  support  resources  should  be  examined  to  determine  Its  Impact 
on  the  prime  hardware  delivery  schedule.  If  the  total  hardware  schedule  Is 
Impacted  by  any  slipped  support  milestones,  the  schedule  requirements  should  be 
evaluated  to  establish  if  the  slippage  is  allowable.  In  the  event  that  the 
total  system/ platform  cannot  stand  the  slippage,  then  the  logistic  support 
resources  delivery  should  be  realigned  to  satisfy  the  prime  equipment. 

5.4  LOGISTIC  SUPPORT  DESIGN  PROCtSS 

For  the  production  phase,  the  five  inputs  to  the  analysis  and  tradeoff 
studies  leading  to  the  fully  supported  system/ platform,  are  outlined  in  Figure  1-1. 
These  input  functions  are  the  end  product  of  development  analyses  which  have 


taken  place  through  the  total  development  process.  Their  relationship  to  the 
total  system/ platform  development  picture  is  that  they  are  the  refined  analyses  of 
the  past  phases  which  have  been  "fine  tuned"  to  meet  the  requirements  of  the 
production  phase.  If  this  is  not  the  case,  then  the  logistic  supporc  design 
functions  have  not  been  adequately  addressed  during  prior  development  phases! 

5.5  ATE  Selection  Process 

5.5.1  General 

Selection  procedures  during  the  production  phase  should  be  minimal  for  on- 
line ATE  and  at  various  levels  for  off-line  AlE.  It  is  not  likely  that  a 
system  should  reach  the  production  phase  of  development  where  the  BIT/BITE 
has  not  been  almost  completely  selected,  but  it  is  possible  to  have  a heavy 
off-line  selection  role.  A candidate  off-line  ATE  selection  situation  could 
occur  when  an  initial  decision  to  use  off-line  ATE  is  made  during  early  develop- 
ment stages,  and  the  decision  is  left  in  a generic  form.  That  Is,  the  ATE 
selection  process  was  truncated  at  a level  where  off-the-shelf,  off-line  ATE 
was  selected  to  perform  the  ATE  support  function,  and  procurement  action 
withheld  until  the  production  phase. 

5.5.2  Selection  Procedure 

5. 5. 2.1  On-Line  ATE 

As  indicated  above,  selection  procedures  for  BIT/BITE  should  be  relegated 
to  selecting  the  design  that  was  proven  In  the  FSDM.  This  design  should  be 
Incorporated  Into  the  pre-production  model  iPPM)  and  subjected  to  the  factory 
acceptance  test.  An  assessment  of  the  ATE  system/ design  should  be  made  after 
the  fabrication  and  test  of  the  PPM. 

5. 5. 2. 2 Off-Line  ATE 

In  contrast  to  the  minimal  on-line  production  phase  selection  procedures, 
there  can  be  possible  heavy  selection  activity  for  off-line  ATE.  These  selection 
procedure  activities  involve  the  following: 

a.  Procurement.  ATE  should  be  procured  to  meet  the  production  per- 
formance specifications  developed  during  the  full  scale  development  phase. 

There  Is  a two  way  path  possible  here  depending  upon  the  contractual  situation 
that  existed  for  the  production  of  the  system/platform.  One  possibility  is 
that  the  prime  production  contractor  has  also  had  full  scale  development  con- 
tract and  the  off-line  ATE  would  be  expected  to  be  the  same  as  the  FSDM  ATE. 

The  other  alternative  is  that  the  off-line  ATE  is  to  be  selected  by  a new 
prime  contractor  for  the  production  phase.  When  this  situation  occurs,  the 
ATE  selection  can  be  subjected  to  "open"  procurement  policy  even  though  past 
development  efforts  made  a "hard"  selection.  To  guard  against  getting  any 
surprises  during  the  production/procurement  phase,  the  procurement  of  new  ATE 
should  be  based  on  the  firm  specifications  developed  from  FSDM  evaluation. 

These  specifications  should  be  In  sufficient  detail  that  all  of  the  functions 
and  capabilities  determined  to  be  of  value  in  the  FSDM  ATE  are  procured  during 
the  production  phase.  Selection  activity  should  be  limited  to  evaluation  of  ATE 
proposals  to  meet  the  specifications. 
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b.  Acceptance  Tests.  Two  actions  should  take  place  at  this  point  In 
the  production  phase;  acceptance  testing  of  the  pre-production  model  and 
development  of  the  TPS  (test  program  sets)  for  the  prime  equipment  UUT's.  The 
results  of  the  acceptance  tests  could  support  the  selection  and  assessment  process. 
TPS  development  should  be  done  in  parallel  to  assure  timely  delivery  of  the 

test  programs  concurrently  with  ATE  availability.  This  Is  possible  for  off-line 
ATE  since  the  UUT's  are  usually  modules  that  are  available  during  construction 
of  the  prime  hardware  and  it  is  not  necessary  to  wait  until  completion  of  the 
total  system  to  start  TPS  development. 

c.  Assessment.  There  are  several  sources  of  Information  during  the 
production  phase  which  can  provide  a basis  for  assessment  of  the  off-line  ATE 
selection.  Feedback  from  the  evaluation  of  proposals,  acceptance  testing  of 
pre-production  model  ATE,  production  line  models,  and  the  success  of  executing 
test  programs  on  the  ATE  can  be  used  to  assess  the  quality  of  the  ATE  selection. 
Assessment  parameters  can  include  the  speed  of  operation  of  the  ATE,  how  well 

it  supports  the  MTTR  requirements,  and  the  percentage  of  required  UUT's  that 
can  be  tested.  Deficiencies  from  the  required  capabilities  which  are  not  due 
to  lack  of  well  defined  specifications  should  be  corrected  by  the  contractor. 

If  the  lack  of  performance  is  due  to  poor  definition  of  the  requirement  or 
specification,  then  additional  funding/contractual  action  is  necessary. 


/)-  23 


6.0  DEPLOYMENT  PHASE 

6.1  General 

In  the  normal  course  of  events,  there  should  be  no  ATE  selection  functions 
after  the  production  phase  has  been  completed.  Certainly,  there  should  be  no 
possible  on-line  BIT/BITE  selection  activity  after  the  full  scale  development 
phase  has  been  completed.  It  Is  quite  possible  that  the  selection  of  off-line 
ATE  could  take  place  In  the  deployment/operational  phase  to  support  depot  enhance- 
ments. There  are  situations  where  ATE  selections  may  be  necessary  due  to  shifts 
in  operational  forces  or  support  concepts.  If  a depot  Is  set  up  or  expanded  to 
respond  to  such  operational  situations,  the  depots  (or  IMA's)  may  require  additional 
ATE  to  perform  new  mission  requirements.  The  selection  process  Is  the  same  as 
those  used  for  the  earlier  development  phases  except  that  the  obvious  tradeoffs 
would  favor  off-the-shelf  ATE  or  Items  available  In  the  Navy  inventory. 


Principal  Efforts 


Once  a system/  pi  at form  is  deployed  and  operational,  the  main  effort  Is  to  keep 
it  fully  logistically  supported  to  a level  where  it  can  meet  the  mission  require- 
ments. Depot  and  IMA  reconfigurations  and  upgrade  actions  also  entail  a signifi- 
cant effort  during  the  deployment  phase.  Additional  test  programs  (TPS)  are  also 
written  to  support  new  UUT' s/modules  (includes  design  changes  which  affect  TPS 
execution)  in  the  deployment/operational  phase.  Modernization  programs  (Fleet 
Modernization  Program)  can  account  for  some  ATE  selection  effort  when  both  the 
platform  and  the  support  system  are  modernized.  It  would  be  expected  that  the 
bulk  of  the  ATE  selection  effort  would  involve  off-line  ATE  rather  than  on-line 
BIT/BITE. 
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7.0  TOOLS,  AIDS  AND  DOCUMENTATION 

7.1  General 


Selection  of  ATE  and  the  counterpart  life  cycle  costing  and  support  system 
tradeoffs  can  be  simplified  through  the  judicious  use  of  available  tools,  aids, 
and  documentation.  The  tools  and  aids  section,  refers  to  computer  implemented 
models,  data  bank  sources,  and  guides  that  would  assist  in  the  selection  of 
ATE  during  various  development  phases.  Documentation  and  references  consist  of 
applicable  DID's  (Data  Item  Descriptions)  and  existing  documentation  applicable 
for  selection  of  ATE. 

7.2  Tools  & Aids  for  Use  in  the  ATE  Selection  Process 


In  Table  7-1,  the  most  applicable  guides,  models,  and  data  banks  are  listed 
by  title,  function,  applicable  life  cycle  phase,  reference  (source  or  authority) 
and  additional  sources  of  information.  The  life  cycle  column  applies  to  the 
development  or  acquisition  phase  outlined  in  Figure  1-1,  and  described  in 
sections  2 through  6 of  this  document.  These  tools  were  identified  in  1974  and 
were  considered  most  applicable  to  the  ATE  selection  process  at  the  time,  and 
are  not  intended  as  a compendium  of  all  possible  sources  of  aid.  If  other  tools 
or  selection  aids  are  considered  by  the  cognizant  ATE  selection  specialists  as 
being  more  suitable,  then  they  should  be  used  instead  of  those  presented  here. 

It  is  requested  that  any  other  models  or  aids  which  are  found  to  be  of  value  in 
the  ATE  selection  process,  and  the  related  support  areas,  should  be  forwarded  to 
the  NAVMAT  Office  (MAT  03T)  responsible  for  this  document.  When  sufficient 
updating  information  is  available,  then  the  following  charts  will  be  updated. 

Data  Item  Descriptions  (DID^)  are  listed  in  Table  7-2  as  a time  saving 
measure  to  assist  acquisition  managers  in  preparation  of  contract  data  require- 
ment lists  (DD  1 423 * s ) for  ATE.  This  list  of  DID's  is  the  result  of  a screening 
of  all  the  DID's  on  file  at  NELC  (Code  4100)  in  mid  1974.  More  suitable  DID's 
may  be  substituted  or  added  as  time  progresses  and  others  are  available  and 
determined  to  be  applicable. 

Additional  information  on  references  in  Tables  7-1  and  7-2,  may  be  obtained 
from  the  Test  Equipment  Technical  Support  Office,  NELC,  Code  4050. 
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TABLE  7-2 


DIO'S  APPLICABLE  TO  ATE 


NUMBER 

COG. 

DI-L-2082 /UDI-R-2 12  28 

NAVY 

DI-L-2083/UDI-R-21229 

NAVY 

DI-L-208A /UDI-R-2 12  30 

NAVY 

DI-L-2085/UDI-R-2131 

NAVY 

DI-S-6171 

ARMY 

DI-S-6168 

ARMY 

DI-S-6170 

rtEMY 

DI-L-6138 

NAVY 

DI-T-3734 

USAF 

ODI-T-22732 

NAVY 

DDI-T-22735B 

NAVY 

TITLE  ? 

LOR  Summary  Report  i 

LOR  Station  Reports  \ 

LOR  Program  Plan  j 

LOR  Analysis  Repairs 

MEA  Data  j 

MEA  Program  Plan  \ 

ILS  Verification  1 

Demonstration  & Evaluation 
Plan 

Integrated  Support  Plan 

Test  Requirements 
Documents 

Test  Point  Measurement 
Parameters  & Recommended 
Support  Test  Equipment  Dat4 


Planned,  Built  In  Test 
Equipment  (BITE)  Fault 
Location 


8.0  SUMMARY 


8.1  Conclusions 

The  most  important  conclusions  to  be  drawn  from  this  report  are: 

a.  Selection  of  ATE  is  an  integral  part  of  prime  system/platform  design. 

b.  Advanced  Development/Validation  phase  Is  the  most  critical  phase  of 
the  ATE  selection  process. 

c.  The  selection  of  ATE  is,  in  most  cases,  an  integral  part  of  the  ILS/LSA 
process. 

8.2  Follow-On  Efforts 


There  are  no  plans  to  issue  this  document  as  a NAVMAT  guide  for  selection 
of  ATE.  This  selection  procedure  will  be  incorporated  into  a variety  of 
existing  and  new  documentation,  which  will  be  an  integral  part  of  prime  hardware 
acquisition  process.  For  example,  the  following  documents  will  be  prepared  or 
revised  to  include  elements  of  the  ATE  selection  procedure: 

a.  A NAVMATINST  on  ATE  data  banks  and  their  uses. 

b.  A Navy  ATE  acquisition  guide. 

c.  A Navy  built-in  test  design  guide. 

d.  Formats  of  other  applicable  documents: 

• Development  Proposal 

• ILS  Plans 

• LSA  R 

• MIL-STD-1 388 

• NAVMATINST  4000. 20A 

• MIL-STD-1 390 

• Data  Items  Descriptions 

In  addition,  the  tools  and  aids  cited  with  be  analyzed  to  ascertain  their  use- 
fulness, and  areas  where  no  aids  exist  will  be  identified. 


In  the  meantime,  use  of  this  document  as  a guide  to  the  selection  of  ATE  is 
encouraged. 


